[Bug][Spark] Positional dynamic partition writes can silently misalign columns

[sablejade]

[Bug][Spark] Positional dynamic partition writes can silently misalign columns

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

Apache Paimon master at commit 437bcc3c84a10b71b827a0a00c1661ccaad6e64f.

Compute Engine

Apache Spark 3.4 with the Paimon Spark SQL Extension.

Minimal reproduce step

Build the paimon-spark-3.4_2.12 bundle from the pinned commit, then start Spark SQL with that bundle and the following configuration:

$SPARK_HOME/bin/spark-sql \
  --jars /path/to/paimon-spark-3.4_2.12-2.1-SNAPSHOT.jar \
  --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
  --conf spark.sql.sources.partitionOverwriteMode=dynamic \
  --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
  --conf spark.sql.catalog.paimon.warehouse=file:///tmp/paimon-dynamic-partition-repro

The reproduction does not require spark.paimon.write.use-v2-write=true. The affected analyzer path is reachable with both the default value, false, and the explicit value, true.

Create a Paimon namespace and a partitioned table:

CREATE NAMESPACE IF NOT EXISTS paimon.repro;
USE paimon.repro;

CREATE TABLE target (
    ds STRING,
    part STRING,
    uid STRING,
    key_name STRING,
    field_name STRING,
    value STRING,
    ttl STRING
)
USING paimon
PARTITIONED BY (ds, part);

Run this positional dynamic partition overwrite:

INSERT OVERWRITE target PARTITION (ds, part)
SELECT
    '20260810' AS ds,
    'p1' AS part,
    'user-1' AS uid,
    'behavioral' AS key_name,
    'metric-a' AS field_name,
    '0.5' AS detail_ratio,
    '3600' AS ttl

UNION ALL

SELECT
    '20260810' AS ds,
    'p2' AS part,
    'user-2' AS uid,
    'behavioral' AS key_name,
    'metric-b' AS field_name,
    '1' AS value,
    '3600' AS ttl;

Read the result:

SELECT ds, part, uid, key_name, field_name, value, ttl
FROM target
ORDER BY ds, part, uid, key_name, field_name, value, ttl;

Expected rows:

ds	part	uid	key_name	field_name	value	ttl
20260810	p1	user-1	behavioral	metric-a	0.5	3600
20260810	p2	user-2	behavioral	metric-b	1	3600

Actual rows on the affected implementation:

ds	part	uid	key_name	field_name	value	ttl
0.5	3600	20260810	p1	user-1	behavioral	metric-a
1	3600	20260810	p2	user-2	behavioral	metric-b

What doesn't meet your expectations?

A positional insert whose output is already in target-table order must not be interpreted as Hive-tail order only because one expression name differs from the target column name.

The current behavior can silently corrupt data when the source and target types are compatible. The analyzer should preserve the table-ordered input in this case while continuing to support genuine Hive-style dynamic partition writes.

Anything else?

Root cause

The affected marker and rewrite path was introduced by #8414 in commit ce92c8a196. It is included in Apache Paimon 2.0.0 and current master.

The failure requires all of the following:

1. The insert has dynamic columns in PARTITION (...), has no user-specified target column list, and is not a BY NAME write.
2. The query is already in target-table position order, but at least one output name differs from the corresponding target name.
3. The computed Hive-tail layout differs from the target table layout. If the dynamic partition columns are already at the end of the table schema, the existing second guard prevents the extra reorder.

The reproduction deliberately declares ds and part first, so the two layouts are:

table:     [ds, part, uid, key_name, field_name, value, ttl]
Hive-tail: [uid, key_name, field_name, value, ttl, ds, part]

The parser rule MarkHiveDynamicPartitionWrite marks every matching insert without checking whether the query is already in table order:

insert.userSpecifiedCols.isEmpty &&
!isByName(insert) &&
insert.partitionSpec.exists(_._2.isEmpty)

It wraps the query in PaimonHiveDynamicPartitionQuery. See AbstractPaimonSparkSqlExtensionsParser.scala.

Later, PaimonAnalysis.resolveDynamicPartitionWrite applies the Hive-style rewrite when both name checks fail:

case Some(hiveStyleOutput)
    if !sameOutputNames(query.output, table.output) &&
      !sameOutputNames(hiveStyleOutput, table.output) =>

See PaimonAnalysis.scala.

In this reproduction:

• sameOutputNames(query.output, table.output) is false because detail_ratio != value.
• sameOutputNames(hiveStyleOutput, table.output) is false because ds and part are declared at the beginning of the table rather than at the end.

The analyzer therefore treats the already table-ordered query as:

[uid, key_name, field_name, value, ttl, ds, part]

Resulting column mapping

The first positional resolution interprets the query as follows:

Query field	Interpreted as
ds	uid
part	key_name
uid	field_name
key_name	value
field_name	ttl
detail_ratio	ds
ttl	part

The subsequent by-name resolution produces:

Target field	Value taken from
ds	detail_ratio
part	ttl
uid	ds
key_name	part
field_name	uid
value	key_name
ttl	field_name

Why UNION exposes the problem

UNION ALL does not reorder the fields and does not create the Hive dynamic-partition marker. Its role is to preserve the correct position order while taking the output names from the first branch.

The second branch alias AS value cannot change the UNION output name inherited from AS detail_ratio in the first branch. This makes the full-name table-order check fail.

The resolved UNION output in the reproduction is therefore:

[ds, part, uid, key_name, field_name, detail_ratio, ttl]

UNION is not required. A single SELECT can trigger the same behavior if its output is already in table order but one output name differs.

Affected write paths

The parser marker can reach:

• AppendData;
• OverwriteByExpression;
• OverwritePartitionsDynamic.

With spark.paimon.write.use-v2-write=false, OverwritePartitionsDynamic is first converted to PaimonDynamicPartitionOverwriteCommand. That fallback command still implements V2WriteCommand, keeps the marked query as its child, and is matched by PaimonAnalysis again on a later analyzer pass. Therefore both values of use-v2-write are affected.

See PaimonDynamicPartitionOverwriteCommand.scala and the write-rule ordering in PaimonAnalysis.scala.

A dynamic OverwritePartitionsDynamic write without a PARTITION (...) clause does not exhibit this alias-driven misalignment at the baseline. That path only selects the Hive layout when the query output names already match the Hive-tail layout; otherwise it retains positional table order.

Proposed fix direction

The minimal fix is to improve the existing automatic decision before applying the Hive-tail rewrite:

1. Require the query and target table to have the same arity.
2. Find the dynamic partition columns and their positions in the target table.
3. If those positions in the query already carry the matching dynamic partition column names, treat the query as table-ordered and do not apply the Hive-tail rewrite.
4. Otherwise, retain the current Hive-tail behavior.

For this reproduction, query.output(0) = ds and query.output(1) = part, which proves that the dynamic partition columns are already at their target-table positions even though the unrelated detail_ratio expression has a different name.

This automatic check cannot resolve every positional ambiguity. For example, if the target partition columns are ds and part but the query fields at those positions are still named dt and pt, names alone cannot prove whether the input is table-ordered or Hive-ordered.

If maintainers want an explicit escape hatch for such ambiguous inputs, an optional configuration could expose:

spark.paimon.sql.dynamic-partition-column-order=AUTO|TABLE|HIVE

• AUTO: use the improved automatic detection and remain the default.
• TABLE: interpret query.output(i) as table.output(i).
• HIVE: require the dynamic partition columns at the end of the query.
• INSERT ... BY NAME: do not apply this positional policy.

The explicit configuration is not required for the minimal fix.

Suggested test coverage

1. A UNION whose first branch has a non-partition alias mismatch while positions follow table order.
2. A single SELECT with the same table-order condition.
3. A genuine Hive-tail dynamic partition write remains supported.
4. A table whose dynamic partition columns are already trailing is not reordered again.
5. AppendData, OverwriteByExpression, and OverwritePartitionsDynamic, including dynamic overwrite with both values of spark.paimon.write.use-v2-write.
6. A no-PARTITION (...) OverwritePartitionsDynamic write and INSERT ... BY NAME remain unaffected.
7. If the optional mode is adopted: explicit TABLE, explicit HIVE, and invalid configuration values.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[sablejade]

If the proposed direction looks reasonable, could you assign this issue to me?
I can submit a PR with the implementation and regression tests.