feat: Update to Calcite 1.41.0.#19370
Conversation
Main changes: 1) Refreshed Parser.jj using dev/upgrade-calcite-parser. 2) Added DruidTypeCoercion to ensure array comparison works properly. We need to be able to coerce BOOLEAN arrays to numeric arrays. 3) Update various toDruidQueryForExplaining implementations to use unique field names for their row signatures. This was a pre-existing bug that was triggered by new Calcite behavior. 4) Update DruidJoinRule to add null gates when hoisting literals over full or left outer joins. This was a pre-existing bug that was triggered by new Calcite behavior. 5) Update FilterDecomposeConcatRule to handle SEARCH. 6) Add InlineValuesSubQueryRule, which is needed to avoid unplannable plans for certain queries mixing UNNEST and SELECT-FROM-VALUES. 7) Add DruidStripUnionArmCastRule, which is needed to remove new CASTs that would otherwise prevent table-level unions from planning. 8) Remove workarounds and negative tests for CALCITE-6435, CALCITE-6715, CALCITE-6435, and CALCITE-6473. 9) Add test to verify WEEK and QUARTER math work properly (CALCITE-6581).
github-actions
Bot
added
Area - Batch Ingestion
Area - Querying
Area - Dependencies
Area - MSQ
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| private DruidStripUnionArmCastRule() | ||
| { | ||
| super(operand(Union.class, any())); |
|
|
||
| private DruidStripUnionArmCastRule() | ||
| { | ||
| super(operand(Union.class, any())); |
| */ | ||
| public static RowSignature fromRelDataTypeWithUniqueFields(final RelDataType rowType) | ||
| { | ||
| return fromRelDataType(SqlValidatorUtil.uniquify(rowType.getFieldNames()), rowType); |
|
This will fix #18665 as well. |
| final List<RelNode> newInputs = new ArrayList<>(union.getInputs().size()); | ||
| boolean anyChanged = false; | ||
| for (final RelNode input : union.getInputs()) { | ||
| final RelNode arm = input.stripped(); |
There was a problem hiding this comment.
P1 Stripping union-arm casts can change native output types
stripArmCasts removes numeric casts from UNION ALL arms whenever the resulting union row type still equals the original row type. That planner-level equality does not guarantee the native Druid query still emits the same column type: removing a BIGINT-to-FLOAT or DECIMAL-to-DOUBLE cast lets the scan arm expose its underlying native type, and the union datasource can then mix arm output types or choose a different runtime type than the SQL row type promised. This can change query results or fail during native union execution. Keep the casts in a project above each arm, or prove the native union path preserves the SQL cast semantics before stripping them.
3 participants
Main changes:
Refreshed Parser.jj using dev/upgrade-calcite-parser.
Added DruidTypeCoercion to ensure array comparison works properly. We need to be able to coerce BOOLEAN arrays to numeric arrays.
Update various toDruidQueryForExplaining implementations to use unique field names for their row signatures. This was a pre-existing bug that was triggered by new Calcite behavior.
Update DruidJoinRule to add null gates when hoisting literals over full or left outer joins. This was a pre-existing bug that was triggered by new Calcite behavior.
Update FilterDecomposeConcatRule to handle SEARCH.
Add InlineValuesSubQueryRule, which is needed to avoid unplannable plans for certain queries mixing UNNEST and SELECT-FROM-VALUES.
Add DruidStripUnionArmCastRule, which is needed to remove new CASTs that would otherwise prevent table-level unions from planning.
Remove workarounds and negative tests for CALCITE-6435, CALCITE-6715, and CALCITE-6473.
Add test to verify WEEK and QUARTER math work properly (CALCITE-6581).