change arrayIngestMode default to array #16789
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
clintropolis
added
Release Notes
Area - Ingestion
Area - MSQ
LakshSingla
reviewed
Jul 24, 2024
docs/querying/arrays.md
Outdated
| as regular scalar strings. SQL `BIGINT ARRAY` and `DOUBLE ARRAY` cannot be loaded under `arrayIngestMode: mvd`. This | ||
| mode is not recommended and will be removed in a future release, but provided for backwards compatibility. | ||
|
|
||
| When `arrayIngestMode` is `none`, Druid throws an exception when trying to store any type of arrays. This mode is most |
There was a problem hiding this comment.
Since we have validations to prevent user error associated with this transition, I wonder if we should remove arrayIngestMode = none.
There was a problem hiding this comment.
makes sense to me to drop it since I can't think of a very strong use case to keep it around.
gianm
approved these changes
Jul 24, 2024
docs/querying/arrays.md
Outdated
| validation unless the column is named under the `skipTypeVerification` context parameter. This parameter can be either | ||
| a comma-separated list of column names, or a JSON array in string form. This validation is done to prevent accidentally | ||
| mixing arrays and multi-value strings in the same column. | ||
| Arrays can be inserted with [SQL-based ingestion](../multi-stage-query/index.md) . |
LakshSingla
approved these changes
Jul 24, 2024
asdf2014
approved these changes
Jul 25, 2024
sreemanamala
pushed a commit
to sreemanamala/druid
that referenced
this pull request
Aug 6, 2024
* change arrayIngestMode default to array * remove arrayIngestMode flag option none * fix space * fix test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The validation added in #15920 should make this relatively safe to do. Some ingestions that are based on the old 'mvd' mode syntax which implicitly changed
ARRAY<STRING>types toSTRINGmight fail, but there is a descriptive error message and the fix is pretty easy.Release note
MSQ ingestion context flag
arrayIngestModenow defaults toarrayinstead ofmvd. This means that SQLVARCHAR ARRAYtypes will no longer be implicitly translated and stored intoVARCHARcolumns, but will instead be stored asVARCHAR ARRAY. Additionally it permits other array types such asBIGINT ARRAYandDOUBLE ARRAYto be inserted with MSQ into their respective array column types (instead of failing as they do inmvdmode). To continue to store multi-value strings modify any insert/replace queries to wrap the array types with theARRAY_TO_MVoperator. Validation is in place that will prevent mixingVARCHARandVARCHAR ARRAYcolumns in the same table, so any ingestions affected by this change will fail and provide a descriptive error message instead of silently doing something unexpected. See https://druid.apache.org/docs/latest/querying/multi-value-dimensions#sql-based-ingestion for additional information about how to ingest multi-value strings, and https://druid.apache.org/docs/latest/querying/arrays#sql-based-ingestion for additional information about arrays.Additionally
arrayIngestModeoption ofnonehas been removed. It was introduced prior to the table validation logic as a means for cluster operators to force query writers to explicitly setarrayormvdon their query contexts, but provides little utility in Druid 31.This PR has: