MSQ arrayIngestMode to control if arrays are ingested as ARRAY, MVD, or an exception by LakshSingla · Pull Request #15093 · apache/druid

LakshSingla · 2023-10-05T06:58:31Z

Description

MSQ uses the string dimension schema for ARRAY<STRING> typed columns, which creates MVDs instead of string arrays as is correct. Therefore someone trying to ingest columns of type ARRAY<STRING> from an external data source or another data source would get STRING columns in the newly generated segments.

This patch adds a arrayIngestMode query context parameter with the following behavior:

If mode is array, uses auto dimension schema to correctly ingest the ARRAY<STRING> columns as ARRAY<STRING> columns, and also allows numeric ARRAY types to be ingested
If mode is mvd, currently default to ease the transition, ARRAY<STRING> will continue to ingest as STRING typed MVDs, but logs will warn operators that this is a deprecated mode. Numeric arrays cannot be ingested in this mode.
If mode is none, neither string or numeric ARRAY types can be ingested, this mode will be used as a forcing mechanism to force people to choose if they explicitly want MVDs or ARRAY typed columns, and in either case, suggest setting the mode to array and explicitly use ARRAY_TO_MV if users still want MVDs.

Release note

MSQ supports a new array mode arrayIngestMode which specifies the behavior of MSQ while ingesting arrays. Please refer to the docs in the release for a detailed behavior of what each option specifies. The default 'mvd' is the existing behaviour of MSQ, therefore doesn't require immediate intervention, however it is subject to removal in future releases.

Key changed/added classes in this PR

MyFoo
OurBar
TheirBaz

This PR has:

gianm · 2023-10-05T14:00:52Z

This change is important but we need to do it less disruptively. People are using MV_TO_ARRAY because they need it for grouping on MVs during ingestion, and our docs recommend using it there. It's a relatively common pattern, and all these people would start getting string arrays instead of MVDs, which will break their queries.

For this reason the change needs to be opt-in. I suggest documenting the context parameter and swapping the default so the old behavior is retained. In addition I'd suggest writing up a doc page about how people can migrate from MVDs to string arrays (with examples of how to rewrite queries) and pointing people at that in the docs for this parameter. And showing people how to use ARRAY_TO_MV to keep things as MVDs. That sets us up for a future where we can deprecate the parameter and swap the default.

gianm · 2023-10-05T14:01:54Z

Another thought: is it possible to add explicit dimension schemas for the various types that can be generated by "auto"? In MSQ we know the exact type we want so it seems odd & circuitous to use "auto".

clintropolis · 2023-10-05T16:35:49Z

Another thought: is it possible to add explicit dimension schemas for the various types that can be generated by "auto"? In MSQ we know the exact type we want so it seems odd & circuitous to use "auto".

I've started doing some work on this, but its sort of non-trivial and shouldn't be part of this PR, so using 'auto' is currently the only way to ingest array columns

clintropolis

lgtm 👍

Are there any tests for trying to insert numeric arrays in mvd mode or any type of arrays in none mode?

abhishekagarwal87 · 2023-10-09T05:01:48Z

...s-core/multi-stage-query/src/main/java/org/apache/druid/msq/util/MultiStageQueryContext.java

+  public static final boolean DEFAULT_USE_AUTO_SCHEMAS = false;
+
+  public static final String CTX_ARRAY_INGEST_MODE = "arrayIngestMode";
+  public static final ArrayIngestMode DEFAULT_ARRAY_INGEST_MODE = ArrayIngestMode.NONE;


Default should be MVD.

cryptoe

Please add the document changes for this as well.

cryptoe · 2023-10-09T05:01:13Z

...s-core/multi-stage-query/src/main/java/org/apache/druid/msq/util/MultiStageQueryContext.java

+  public static final boolean DEFAULT_USE_AUTO_SCHEMAS = false;
+
+  public static final String CTX_ARRAY_INGEST_MODE = "arrayIngestMode";
+  public static final ArrayIngestMode DEFAULT_ARRAY_INGEST_MODE = ArrayIngestMode.NONE;


The default mode should be MVD IMHO since it will not break stuff.

I personally disagree, I think we should default to none, then people explicitly choose to use MVDs or arrays

mostly because the behavior of MVD is totally incorrect

I am working on updating the docs/examples refer to the parameter and to separate them into ones that explicitly store MVDs using ARRAY_TO_MV function, and ones which only use ARRAY_ functions into examples of storing array typed columns instead.

MVD mode is not going to have examples, because it should be the first to go and no one should rely on this behavior, because again the behavior is incorrect.

IMHO we cannot break user's ingestion sql query so I propose the mode should be MVD.
The patch should also throw a warning from the controller and throw it on the console if a row signature with Array is detected. The warning clearly lays out the path of migration.

There are lot of layers build out in organizations. A change even if its adding a context flag to sometimes takes weeks to reach to production. With this warning we effectively nudge the user that we are going to break compatibility soon.

clintropolis · 2023-10-09T05:42:42Z

...ons-core/multi-stage-query/src/main/java/org/apache/druid/msq/util/DimensionSchemaUtils.java

+                    "String arrays can not be ingested when '%s' is set to '%s'. Either set '%s' in query context "
+                    + "to 'array' to ingest the string array as an array, or set it to 'mvd' to ingest the string array "
+                    + "as MVD (which is legacy behaviour and not recommmended)",
+                    MultiStageQueryContext.CTX_ARRAY_INGEST_MODE,
+                    StringUtils.toLowerCase(arrayIngestMode.name()),
+                    MultiStageQueryContext.CTX_ARRAY_INGEST_MODE


I don't think we should recommend MVD mode at all here, instead we should always recommend array mode and suggest using ARRAY_TO_MV if people want to store things as a MVD.

LakshSingla · 2023-10-09T12:23:01Z

Are there any tests for trying to insert numeric arrays in mvd mode or any type of arrays in none mode?

Yes, there are tests verifying that arrays cannot be inserted in none mode, and numeric arrays cannot be inserted in MVD mode.

LakshSingla · 2023-10-10T18:55:57Z

Thanks, @clintropolis for updating & aligning the description with what we actually merged, and not the stale changes.

MSQ uses the string dimension schema for ARRAY<STRING> typed columns, which creates MVDs instead of string arrays as required. Therefore someone trying to ingest columns of type ARRAY<STRING> from an external data source or another data source would get STRING columns in the newly generated segments. This patch changes the following: - Use auto dimension schema to ingest the ARRAY<STRING> columns, which will create columns with the desired type. - Add an undocumented flag ingestStringArraysAsMVDs to preserve the legacy behavior. Legacy behaviour is turned on by default. - Create MSQArraysInsertTest and refactor some of the tests in MSQInsertTest.

init

db2ac3a

github-actions bot added Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Oct 5, 2023

abhishekagarwal87 added the Release Notes label Oct 5, 2023

arrayIngestMode

9fd8258

clintropolis approved these changes Oct 9, 2023

View reviewed changes

abhishekagarwal87 requested changes Oct 9, 2023

View reviewed changes

cryptoe reviewed Oct 9, 2023

View reviewed changes

clintropolis reviewed Oct 9, 2023

View reviewed changes

LakshSingla added 3 commits October 9, 2023 14:14

indentation fix ??

b6bcd4e

change case to if..else for intellij

23effc4

change dim utils because intellij

9da28c4

change default to mvd

fb0e341

LakshSingla requested a review from abhishekagarwal87 October 9, 2023 13:39

abhishekagarwal87 approved these changes Oct 9, 2023

View reviewed changes

abhishekagarwal87 merged commit b0edbc3 into apache:master Oct 9, 2023

clintropolis changed the title ~~MSQ writes out string arrays instead of MVDs by default~~ MSQ arrayIngestMode to control if arrays are ingested as ARRAY, MVD, or an exception Oct 9, 2023

LakshSingla added this to the 28.0 milestone Oct 12, 2023

LakshSingla deleted the msq-write-string-array branch October 13, 2023 19:39

clintropolis mentioned this pull request Oct 25, 2023

explicit outputType for ExpressionPostAggregator, better documentation for the differences between arrays and mvds #15245

Merged

LakshSingla mentioned this pull request Nov 4, 2023

[DRAFT] 28.0.0 release notes #15326

Closed

Comments

Conversation

LakshSingla commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release note

Key changed/added classes in this PR

Uh oh!

gianm commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gianm commented Oct 5, 2023

Uh oh!

clintropolis commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clintropolis left a comment

Choose a reason for hiding this comment

Uh oh!

abhishekagarwal87 Oct 9, 2023

Choose a reason for hiding this comment

Uh oh!

cryptoe left a comment

Choose a reason for hiding this comment

Uh oh!

cryptoe Oct 9, 2023

Choose a reason for hiding this comment

Uh oh!

clintropolis Oct 9, 2023

Choose a reason for hiding this comment

Uh oh!

clintropolis Oct 9, 2023

Choose a reason for hiding this comment

Uh oh!

clintropolis Oct 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cryptoe Oct 9, 2023

Choose a reason for hiding this comment

Uh oh!

clintropolis Oct 9, 2023

Choose a reason for hiding this comment

Uh oh!

LakshSingla commented Oct 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LakshSingla commented Oct 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

LakshSingla commented Oct 5, 2023 •

edited

Loading

gianm commented Oct 5, 2023 •

edited

Loading

clintropolis commented Oct 5, 2023 •

edited

Loading

clintropolis Oct 9, 2023 •

edited

Loading

LakshSingla commented Oct 9, 2023 •

edited

Loading

LakshSingla commented Oct 10, 2023 •

edited

Loading