feat: v3 write API and line protocol validation #25048
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit points at the series key branch on influxdb3_core, and refac- tors the code to make use of it throughout. TableDefinitions in the catalog are updated to support the series key column type, and will cause panics when de/serializing table snapshots that contain cross-contaminated column types, i.e., that have both tags and series key columns. This does not implement the v3 write API or any of the stack that performs the writes to use these new TableDefinition features. Several tests are broken by this commit, so may be fixed in a future commit, or with ammendments, but this is a stopping point before pursuing further changes.
A end-to-end test was added to test writing to the v3 write API, along with tests for failure modes to check the validation of incoming writes.
|
Closed, see #25066 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #25033
Summary
This PR leverages a set of changes introduced experimentally into
influxdb3_coreto enable the v3 line protocol write API proposed in #24979.You can see the result in an end-to-end test here, that performs a write to the
/api/v3/writeAPI, and then performs a set of queries to verify that the data can be queried, and that the new line protocol is validated for correctness in the written series key.Detailed Changes
Extend the Catalog to support tables with series key
After #25031, the
TableDefinitionin the catalog is just a wrapper around theSchematype from the coreschemacrate. Therefore, a lot of the following functionality stems from the changes made to that type to support the series key ininfluxdb3_core.v1/v2and `v3 tables simultaneously; the difference between them is:v3has a schema-level metadata entry that stores the series key membersv3can containKeycolumns, but notTagcolumns, whilev1/v2is the inversev1/v2:v3:Split the write path to enable validation and buffering of
v1andv3writesThe code that parsed and validated incoming writes was previously written as a series of nested function calls. This PR refactored that code by:
WriteValidatorto store the state that is re-used throughout the process. There are three steps to validation:WriteValidatorimplementation to do (i.) and (ii.) forv3Add a new
/api/v3/writeAPIThis is the API that leverages the new write path, and can be used to perform writes using the new write protocol. It accepts the same parameters as the existing
/api/v3/write_lpAPI.