[spark] Fix nested struct values wrong order for insert columns - #9415
[spark] Fix nested struct values wrong order for insert columns#9415ArnavBalyan wants to merge 3 commits into
Conversation
|
cc @JingsongLi thanks :) |
| } | ||
|
|
||
| private def renameFieldsInStruct(input: StructType, expected: StructType): StructType = { | ||
| if (input.length == expected.length) { |
There was a problem hiding this comment.
[P1] Preserve positional mapping when nested field counts differ
Returning input here falls back to the downstream by-name resolver whenever merge-schema fills a missing nested field. For target ARRAY<STRUCT<x: INT, y: INT, z: INT>>, a column-list write whose input struct fields are (y=20, x=10) stores [10, 20, null] instead of the positional [20, 10, null]. I reproduced this on Spark 3.5 with spark.paimon.write.merge-schema=true. Please rename the common ordinal prefix even when the lengths differ, preserve unmatched input fields, and let the existing strict/merge-schema handling process missing or extra fields. A regression test with unequal nested field counts would cover this path.
|
Hi @JingsongLi thanks for the review, have addressed the comment |
| val queryWithoutMarker = stripHiveDynamicPartitionMarker(v2WriteCommand.query) | ||
| val query = | ||
| if ( | ||
| v2WriteCommand.isByName && |
There was a problem hiding this comment.
[P1] Do not gate column-list semantics on Spark\x27s version-dependent isByName
The current head fails its own new Paimon Insert: column list resolves unequal nested structs positionally test on both Spark 3.2 and 3.3 with Scala 2.13. The CI stack reaches resolveColumnsByPosition at arr.element and rejects the 2-field source vs 3-field target; Spark 3.4/3.5 pass. On the older plans this isByName gate does not select the column-list rewrite/merge-schema path, even though the parser marker identifies the same SQL syntax.
Please derive the effective column-list mode from the explicit COLUMN_LIST_WRITE marker independently of Spark\x27s V2WriteCommand.isByName, and use that effective mode consistently for nested renaming, expected-output calculation, and missing-field resolution. The existing test already exposes the regression; it needs to pass under every supported Spark 3 profile before merge.
2 participants
Purpose
Tests