[SPARK-54289][SQL] Allow MERGE INTO to preserve existing struct fields for UPDATE SET * when source struct has less nested fields than target struct #53149
+1,075
−207
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
szehon-ho
changed the title
szehon-ho
force-pushed
the
merge_schema_evolution_update_nested
branch
3 times, most recently
from
db416a9 to
142d795
Compare
…s for UPDATE SET * when source has less fields
szehon-ho
force-pushed
the
merge_schema_evolution_update_nested
branch
from
142d795 to
fdddef1
Compare
Member
There was a problem hiding this comment.
Is this an improvement, @szehon-ho ?
It looks like a massive change if it's a bug fix.
Never mind. I checked the JIRA discussion we had before.
Member
Author
|
@cloud-fan can you help review? Thanks |
szehon-ho
commented
Nov 21, 2025
| } | ||
| } | ||
|
|
||
| private def applyNestedFieldAssignments( |
Member
Author
There was a problem hiding this comment.
Note: this is like applyFieldAssignments above, but recurses into nested structs
| .createWithDefault(true) | ||
|
|
||
| val MERGE_INTO_SOURCE_NESTED_TYPE_UPDATE_BY_FIELD = | ||
| buildConf("spark.sql.merge.nested.type.assign.by.fieldv2") |
Member
There was a problem hiding this comment.
Hi, @szehon-ho . The naming space design looks weird. Why nested is at the different level like the following?
spark.sql.merge.nested.type.assign.by.fieldv2
spark.sql.merge.source.nested.type.coercion.enabled
Member
Author
There was a problem hiding this comment.
ah yes i fixed the config string in latest one, thanks!
| getConf(SQLConf.LEGACY_XML_PARSER_ENABLED) | ||
|
|
||
| def coerceMergeNestedTypes: Boolean = | ||
| def mergeCoerceNestedTypes: Boolean = |
Member
There was a problem hiding this comment.
Oh, you also knew that the naming is weird, don't you, @szehon-ho ? That's the reason you renaming this in this PR.
Member
Author
There was a problem hiding this comment.
ah yes, its not strictly related, but realized its better they align.
dongjoon-hyun
pushed a commit
that referenced
this pull request
Nov 22, 2025
…s for UPDATE SET * when source struct has less nested fields than target struct ### What changes were proposed in this pull request? Introduce a new flag spark.sql.merge.nested.type.assign.by.field that allows UPDATE SET * action in MERGE INTO to be shorthand to assign every nested struct to its existing source counterpart (ie, UPDATE SET a.b.c = source.a.b.c). This will have the implication that existing struct field in the target table that has no source equivalent are preserved, when the corresponding source struct has less fields than target. Additional code is added to prevent null expansion in this case (ie, a null source struct expanding to a struct of nulls). ### Why are the changes needed? Following #52347, we now allow MERGE INTO to have a source table struct with less nested fields than target table struct. In this scenario, a user making a UPDATE SET * may have two interpretations. The use may interpret UPDATE SET * as shorthand to assign every top-column level field, ie UPDATE SET struct=source.struct, then the target struct is set to source struct object as is, with missing fields as NULL. This is the current behavior. The user may also mean that UPDATE SET * is short-hand to assign every nested struct field (ie, UPDATE SET struct.a.b = source.struct.a.b), in which case the target struct fields missing in source are retained. This is similar to UPDATE SET * not overriding existing target columns missing in the source, for example. For this case, this flag is added. ### Does this PR introduce _any_ user-facing change? No, the support to allow source structs to have less fields than target structs in MERGE INTO is unreleased yet (#52347), and in any case there is a flag to toggle this functionality. ### How was this patch tested? Unit tests, especially around cases where the source struct is null. ### Was this patch authored or co-authored using generative AI tooling? No Closes #53149 from szehon-ho/merge_schema_evolution_update_nested. Authored-by: Szehon Ho <szehon.apache@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 966e053) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Member
|
Merged to master/4.1 for Apache Spark 4.1.0. Happy Thanksgiving, @szehon-ho . |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Introduce a new flag spark.sql.merge.nested.type.assign.by.field that allows UPDATE SET * action in MERGE INTO to be shorthand to assign every nested struct to its existing source counterpart (ie, UPDATE SET a.b.c = source.a.b.c). This will have the implication that existing struct field in the target table that has no source equivalent are preserved, when the corresponding source struct has less fields than target.
Additional code is added to prevent null expansion in this case (ie, a null source struct expanding to a struct of nulls).
Why are the changes needed?
Following #52347, we now allow MERGE INTO to have a source table struct with less nested fields than target table struct. In this scenario, a user making a UPDATE SET * may have two interpretations.
The use may interpret UPDATE SET * as shorthand to assign every top-column level field, ie UPDATE SET struct=source.struct, then the target struct is set to source struct object as is, with missing fields as NULL. This is the current behavior.
The user may also mean that UPDATE SET * is short-hand to assign every nested struct field (ie, UPDATE SET struct.a.b = source.struct.a.b), in which case the target struct fields missing in source are retained. This is similar to UPDATE SET * not overriding existing target columns missing in the source, for example. For this case, this flag is added.
Does this PR introduce any user-facing change?
No, the support to allow source structs to have less fields than target structs in MERGE INTO is unreleased yet (#52347), and in any case there is a flag to toggle this functionality.
How was this patch tested?
Unit tests, especially around cases where the source struct is null.
Was this patch authored or co-authored using generative AI tooling?
No