[spark] Unify MERGE INTO assignment alignment

[Zouxxyy]

Purpose

Route MERGE INTO assignment alignment through PaimonOutputResolver so MERGE / UPDATE / INSERT share one name-based, depth-aware alignment path with consistent merge-schema semantics.

Behavior is controlled by spark.paimon.write.merge-schema:

• Top-level alignment is by name. Unmentioned target columns are NULL-filled under explicit clauses (matches Spark INSERT FILL).
• INSERT * / UPDATE * with a source column missing from the target throws when merge-schema=false; when true, source-extras are kept so SchemaHelper evolves the table at write time.
• Nested struct alignment follows MissingFieldBehavior:
  • FailMissing (merge-schema=false): nested missing target / source-extra throws.
  • NullForMissing (merge-schema=true, INSERT and explicit UPDATE): missing nested fields NULL-fill, source-extras kept at any depth.
  • PreserveTarget (merge-schema=true, UPDATE * on a struct target): missing source subfields are substituted with GetStructField(target, ordinal) so unmentioned subfields keep their current value instead of being nulled.

Tests

New MergeIntoAlignmentTest (24 cases) covers basic UPDATE * / INSERT *, source-extra drop under strict star, top-level and nested merge-schema evolution, PreserveTarget semantics on nested struct UPDATE *, explicit assignments to nested fields, and null-fill for omitted columns.

[leaves12138]

I reviewed the MERGE INTO alignment changes. The common alignment/evolution path and the Spark 3/4 shims look reasonable to me, and CI is green. I found one documentation mismatch that should be fixed before merge.

[leaves12138]

This section does not seem to match the implementation and the new tests. For top-level source-only columns under UPDATE SET * / INSERT *, strict mode currently drops them (for example source extra columns silently dropped under star expansion (mergeSchema=false)), while merge-schema=true evolves the target schema. Conversely, for target-only columns missing from the source, strict UPDATE SET * / INSERT * throws, and only merge-schema mode preserves the target value for UPDATE SET * or fills default/NULL for INSERT *. Could you adjust these bullets to describe those four cases explicitly?

[JingsongLi]

This is a significant refactoring with good motivations — unifying the assignment alignment logic removes a lot of duplicated version-specific code. The +1777/-1481 diff shows meaningful consolidation.

A few observations:

1. MissingFieldBehavior semantics are well-thought-out: The three modes (FailMissing, NullForMissing, PreserveTarget) cover the right matrix of explicit vs star clauses under strict/merge-schema modes. The PreserveTarget mode for UPDATE * on nested structs (substituting GetStructField(target, ordinal) for unmentioned subfields) is the correct behavior.

2. Test coverage looks thorough: 24 cases in MergeIntoAlignmentTest covering the key scenarios. This gives me confidence in the refactoring.

3. Question about the Spark 4.0 path: The diff removes PaimonMergeIntoResolverBase.scala from paimon-spark-4.0 and significantly simplifies Spark41MergeIntoRewrite.scala. Can you confirm the Spark 4.0 tests pass? The Spark 4 MERGE INTO semantics differ from 3.x in how they handle star clause expansion.

4. Documentation update: Good that the docs update mentions the spark.paimon.write.merge-schema control. This is user-facing behavior change worth highlighting in release notes.

Overall this looks like a good consolidation. @Zouxxyy please confirm CI passes for all Spark versions (3.2, 3.3, 3.4, 3.5, 4.0).