feature(v3): Write: partial writes write good lines and reject bad li… #5169

jstirnaman · 2023-10-10T23:14:26Z

…nes #5160

Documents the proposed partial writes feature for Serverless.
Updates Troubleshoot writes and API docs.

Closes #5160

wiedld

TY for doing these updates!

Made some suggestions, mostly to:

Help delineate the products (serverless vs dedicated) different options -- because only dedicated will have this be configurable.
Provide two different examples of 400 errors. (Hopefully the user can then "type" the difference.) Altho I'm unclear how best to do this. 🤷🏼‍♀️

api-docs/cloud-dedicated/ref.yml

api-docs/cloud-serverless/ref.yml

content/influxdb/cloud-dedicated/write-data/troubleshoot.md

jstirnaman · 2023-10-11T16:56:54Z

@wiedld Thanks for the helpful suggestions here!

wiedld

Approving.
With the caveat of please wait until change is deployed before merging. 🙏🏼

sanderson

One minor suggestion. Looks great!

content/influxdb/cloud-dedicated/write-data/troubleshoot.md

wiedld

Changes requested based on updated error code. TY!

api-docs/cloud-dedicated/ref.yml

api-docs/cloud-serverless/ref.yml

content/influxdb/cloud-dedicated/write-data/troubleshoot.md

content/influxdb/cloud-serverless/write-data/troubleshoot.md

#5160 - Documents the proposed partial writes feature for Serverless and Dedicated. - If some points are written, returns 201 status code. - Updates Troubleshoot writes and API docs.

jacobmarble

The 201 HTTP response code is not what v2 products return on partial errors. We return 201 because of an idiosyncrasy in an upstream Rust library.

Before we merge this, I'd like to see if we can correct this behavior to avoid (1) customer surprises and (2) more surprises when we eventually fix this.

Relevant comment: https://github.com/influxdata/influxdb_iox/issues/5226#issuecomment-1850759520

jstirnaman · 2023-12-13T17:28:49Z

api-docs/cloud-serverless/ref.yml

+              examples:
+                partialWriteErrorWithRejectedPoints:
+                  summary: Partial write rejects points with syntax errors
+                  value:


@Jayclifford345 201 (partial write) and 400 (batch failed) response body properties.

jstirnaman · 2023-12-13T17:30:38Z

@wiedld @sanderson Please give this Serverless docs PR a look. I moved Dedicated changes to a separate branch.

… codes.

api-docs/cloud-serverless/ref.yml

sanderson

One comment about writing a tag and field with the same key, but nothing that should hold this PR up. Everything else looks great.

sanderson · 2023-12-13T18:12:55Z

content/influxdb/cloud-serverless/write-data/troubleshoot.md

+- an invalid timestamp
+- a schema conflict with existing tags or fields in the database
+
+A schema conflict can occur when points that fall within the same partition (default partitioning is measurement and day) as existing bucket data have a different data type for an existing field.


Points with a tag and field that use the same key will also be rejected. I think these would be considered a "schema conflict," but I don't know if they'll get a line protocol parsing error or something else. Maybe something to note here.

Points with a tag and field that use the same key will also be rejected. I think these would be considered a "schema conflict," but I don't know if they'll get a line protocol parsing error or something else. Maybe something to note here.

Good catch! @wiedld Do you know when duplicate keys get detected? If not, I'll test post-change and update docs.

I believe this should be a schema conflict per line, and therefore the same rules apply (if in partial versus non-partial writes).

api-docs/cloud-serverless/ref.yml

jacobmarble

Please don't merge this.

We shouldn't publicly document that partial write errors return 201.

This is inconsistent with our V2 API.
We will fix this behavior so that 204 is returned instead.
Documenting will only confuse customers going forward.

jacobmarble · 2023-12-13T19:04:05Z

Here's the 201/204 fix for Serverless:
https://github.com/influxdata/idpe/pull/18398

wiedld · 2023-12-13T19:08:29Z

Here's the 201/204 fix for Serverless: influxdata/idpe#18398

We are adding a change for Serverless, to make the 201 => 204.
The dedicated cluster will still have the 204 & 204.

jacobmarble · 2023-12-13T19:09:48Z

The dedicated cluster will still have the 204 & 204.

I don't understand this. Typo?

jstirnaman · 2023-12-13T19:28:41Z

Here's the 201/204 fix for Serverless: influxdata/idpe#18398

@wiedld @jacobmarble To be clear, a partial write response for Serverless will have status 204 No Content, but will have a response body?

- Documents influxdata/idpe#18398.

wiedld · 2023-12-13T21:26:17Z

api-docs/cloud-serverless/ref.yml

@@ -14601,8 +14680,9 @@ tags:
      | &nbsp;Code&nbsp; | Status                   | Description      |
      |:-----------:|:------------------------ |:--------------------- |
      | `200`       | Success                  |                       |
-      | `204`       | Success. No content      | InfluxDB doesn't return data for the request. For example, a successful write request returns `204` status code, acknowledging that data is written and queryable. |
-      | `400`       | Bad request              | InfluxDB can't parse the request due to an incorrect parameter or bad syntax. For _writes_, the error may indicate one of the following problems: <ul><li>Line protocol is malformed. The response body contains the first malformed line in the data and indicates what was expected. For partial writes, the number of points written and the number of points rejected are also included.</li><li>`Authorization` header is missing or malformed or the API token doesn't have permission for the operation.</li></ul> |
+      | `201`       | Created                           | Successfully created a resource. The response body may contain details. |


With Jacob's change, the 201 will not exist in serverless (but will still exist in dedicated).

wiedld · 2023-12-13T21:27:09Z

content/influxdb/cloud-serverless/write-data/troubleshoot.md

-| `400 "Bad request"`             | `message` contains the first malformed line                             | If request data is malformed |
+| HTTP response code              | Response body                                                                    | Description    |
+| :-------------------------------| :---------------------------------------------------------------        | :------------- |
+| `201 "Created"`                 | error details about rejected points, up to 100 points: `line` contains the first rejected line, `message` describes rejected points                                                                     | If InfluxDB ingested some or all of the data |


content/influxdb/cloud-serverless/write-data/troubleshoot.md

…t bad li… (#5169)" This reverts commit 90948e6.

…t bad li… (#5169)" (#5280) This reverts commit 90948e6.

…nd reject bad li… (#5169)" (#5280)" This reverts commit 624eb02.

…nd reject bad li… (#5169)" (#5280)" (#5304) This reverts commit 624eb02.

jstirnaman requested review from sanderson and wiedld October 10, 2023 23:14

wiedld reviewed Oct 11, 2023

View reviewed changes

jstirnaman mentioned this pull request Oct 11, 2023

Can we have 2 examples? One for rejected points error, and another for an entire rejection (e.g. the namespace error). #5170

Closed

jstirnaman mentioned this pull request Oct 11, 2023

Create schema and samples for non-400 write request errors #5171

Open

wiedld approved these changes Oct 12, 2023

View reviewed changes

sanderson approved these changes Oct 13, 2023

View reviewed changes

content/influxdb/cloud-dedicated/write-data/troubleshoot.md Outdated Show resolved Hide resolved

wiedld requested changes Dec 13, 2023

View reviewed changes

jstirnaman added 2 commits December 13, 2023 11:06

feature(v3): Serverless write API: writes good lines, rejects bad lines

489f196

#5160 - Documents the proposed partial writes feature for Serverless and Dedicated. - If some points are written, returns 201 status code. - Updates Troubleshoot writes and API docs.

feature(v3): Serverless write API: writes good lines, rejects bad lines

303a161

#5160 - Documents the proposed partial writes feature for Serverless and Dedicated. - If some points are written, returns 201 status code. - Updates Troubleshoot writes and API docs.

jstirnaman force-pushed the feature-5160-partial-writes branch from 0555a3f to 303a161 Compare December 13, 2023 17:22

fix: Serverless cleanup

3aedab1

jacobmarble requested changes Dec 13, 2023

View reviewed changes

jstirnaman commented Dec 13, 2023

View reviewed changes

jstirnaman marked this pull request as ready for review December 13, 2023 17:29

jstirnaman requested review from jacobmarble, sanderson and wiedld December 13, 2023 17:31

fix: Serverless write API: restore 204 and explain 201 and 400 status…

1f27555

… codes.

wiedld approved these changes Dec 13, 2023

View reviewed changes

api-docs/cloud-serverless/ref.yml Show resolved Hide resolved

sanderson approved these changes Dec 13, 2023

View reviewed changes

jstirnaman commented Dec 13, 2023

View reviewed changes

api-docs/cloud-serverless/ref.yml Outdated Show resolved Hide resolved

Apply suggestions from code review

e3c14c0

jacobmarble requested changes Dec 13, 2023

View reviewed changes

jstirnaman added 2 commits December 13, 2023 15:02

fix: Serverless write API: 201 superseded by 204 to maintain v2 compat.

2b6c563

- Documents influxdata/idpe#18398.

fix: Serverless write API: 201 superseded by 204 to maintain v2 compat.

c9a48dc

- Documents influxdata/idpe#18398.

wiedld reviewed Dec 13, 2023

View reviewed changes

jstirnaman commented Dec 13, 2023

View reviewed changes

content/influxdb/cloud-serverless/write-data/troubleshoot.md Outdated Show resolved Hide resolved

jstirnaman commented Dec 13, 2023

View reviewed changes

content/influxdb/cloud-serverless/write-data/troubleshoot.md Outdated Show resolved Hide resolved

jstirnaman commented Dec 13, 2023

View reviewed changes

content/influxdb/cloud-serverless/write-data/troubleshoot.md Outdated Show resolved Hide resolved

jstirnaman added 3 commits December 13, 2023 16:05

Update content/influxdb/cloud-serverless/write-data/troubleshoot.md

24ce614

Update content/influxdb/cloud-serverless/write-data/troubleshoot.md

253e3af

Update content/influxdb/cloud-serverless/write-data/troubleshoot.md

23c94da

jstirnaman merged commit 90948e6 into master Dec 13, 2023
2 checks passed

jstirnaman deleted the feature-5160-partial-writes branch December 13, 2023 22:20

jstirnaman added a commit that referenced this pull request Dec 15, 2023

Revert "feature(v3): Write: partial writes write good lines and rejec…

92f9bed

…t bad li… (#5169)" This reverts commit 90948e6.

jstirnaman mentioned this pull request Dec 15, 2023

Revert "feature(v3): Write: partial writes write good lines and reject bad li…" #5280

Merged

jstirnaman added a commit that referenced this pull request Dec 15, 2023

Revert "feature(v3): Write: partial writes write good lines and rejec…

624eb02

…t bad li… (#5169)" (#5280) This reverts commit 90948e6.

jstirnaman added a commit that referenced this pull request Jan 17, 2024

Revert "Revert "feature(v3): Write: partial writes write good lines a…

a7b96d0

…nd reject bad li… (#5169)" (#5280)" This reverts commit 624eb02.

jstirnaman added a commit that referenced this pull request Jan 17, 2024

Revert "Revert "feature(v3): Write: partial writes write good lines a…

7479cea

…nd reject bad li… (#5169)" (#5280)" (#5304) This reverts commit 624eb02.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature(v3): Write: partial writes write good lines and reject bad li… #5169

feature(v3): Write: partial writes write good lines and reject bad li… #5169

jstirnaman commented Oct 10, 2023 •

edited

Loading

wiedld left a comment •

edited

Loading

jstirnaman commented Oct 11, 2023

wiedld left a comment

sanderson left a comment

wiedld left a comment

jacobmarble left a comment

jstirnaman Dec 13, 2023

jstirnaman commented Dec 13, 2023

sanderson left a comment

sanderson Dec 13, 2023 •

edited

Loading

jstirnaman Dec 13, 2023

wiedld Dec 13, 2023

jacobmarble left a comment

jacobmarble commented Dec 13, 2023

wiedld commented Dec 13, 2023 •

edited

Loading

jacobmarble commented Dec 13, 2023

jstirnaman commented Dec 13, 2023 •

edited

Loading

wiedld Dec 13, 2023

wiedld Dec 13, 2023

feature(v3): Write: partial writes write good lines and reject bad li… #5169

feature(v3): Write: partial writes write good lines and reject bad li… #5169

Conversation

jstirnaman commented Oct 10, 2023 • edited Loading

wiedld left a comment • edited Loading

Choose a reason for hiding this comment

jstirnaman commented Oct 11, 2023

wiedld left a comment

Choose a reason for hiding this comment

sanderson left a comment

Choose a reason for hiding this comment

wiedld left a comment

Choose a reason for hiding this comment

jacobmarble left a comment

Choose a reason for hiding this comment

jstirnaman Dec 13, 2023

Choose a reason for hiding this comment

jstirnaman commented Dec 13, 2023

sanderson left a comment

Choose a reason for hiding this comment

sanderson Dec 13, 2023 • edited Loading

Choose a reason for hiding this comment

jstirnaman Dec 13, 2023

Choose a reason for hiding this comment

wiedld Dec 13, 2023

Choose a reason for hiding this comment

jacobmarble left a comment

Choose a reason for hiding this comment

jacobmarble commented Dec 13, 2023

wiedld commented Dec 13, 2023 • edited Loading

jacobmarble commented Dec 13, 2023

jstirnaman commented Dec 13, 2023 • edited Loading

wiedld Dec 13, 2023

Choose a reason for hiding this comment

wiedld Dec 13, 2023

Choose a reason for hiding this comment

jstirnaman commented Oct 10, 2023 •

edited

Loading

wiedld left a comment •

edited

Loading

sanderson Dec 13, 2023 •

edited

Loading

wiedld commented Dec 13, 2023 •

edited

Loading

jstirnaman commented Dec 13, 2023 •

edited

Loading