branch-4.1: [feature](paimon) Support row-level UPDATE, DELETE, and MERGE

[suxiaogang223]

What changed

This PR adds row-level DML support for Apache Paimon tables through the Doris Paimon catalog:

• UPDATE ... SET ... WHERE ...
  • Supports expressions in assignments and predicates.
  • Supports primary-key tables using the deduplicate and partial-update merge engines.
  • Rejects updates to primary-key columns.
• DELETE FROM ... WHERE ...
  • Supports primary-key tables using deduplicate.
  • Supports partial-update and aggregate tables when their Paimon delete/removal options are enabled.
• MERGE INTO ... USING ... ON ...
  • Supports conditional WHEN MATCHED THEN UPDATE.
  • Supports conditional WHEN MATCHED THEN DELETE.
  • Supports conditional WHEN NOT MATCHED THEN INSERT.
  • Supports combining update, delete, and insert branches in one statement.

Append-only tables are rejected for row-level DML. Unsupported merge engines and table options fail during analysis with explicit error messages.

Implementation

• Adds dedicated Nereids commands for Paimon UPDATE, DELETE, and MERGE.
• Carries a per-row operation value through the logical and physical Paimon sink.
• Converts the operation value into Paimon row kinds in the JNI writer.
• Binds changelog sink outputs at the logical sink level, preserving the operation column while coercing data columns to the target schema.
• Adds target-table collection and schema-change retry integration for bound Paimon row-level sinks.
• Adds regression coverage for successful UPDATE, DELETE, and mixed MERGE operations, plus unsupported append-only and first-row cases.

User impact

Users can modify existing Paimon primary-key tables directly with standard Doris SQL instead of rewriting the table through INSERT or an external compute engine.

Validation

• BUILD_TYPE=Debug DISABLE_BUILD_UI=ON ./build.sh --fe -j12
• Debug BE build using the Doris toolchain
• test_paimon_write_row_level_dml: UPDATE, DELETE, MERGE, and negative cases all passed
• FE and BE deployment health checks passed

Related to #65086

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[suxiaogang223]

run buildall

[hello-stephen]

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.62% (1904/2453)
Line Coverage	64.46% (34018/52774)
Region Coverage	64.46% (17208/26694)
Branch Coverage	53.92% (9201/17064)

[hello-stephen]

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	58.27% (24674/42347)
Line Coverage	42.51% (248497/584589)
Region Coverage	38.57% (196304/508963)
Branch Coverage	39.94% (89813/224850)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.75% (30440/41275)
Line Coverage	57.70% (334767/580214)
Region Coverage	54.60% (278665/510329)
Branch Coverage	55.52% (124692/224608)

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 2.38% (9/378) 🎉
Increment coverage report
Complete coverage report

[Gabriel39]

Thanks for adding Paimon row-level DML support. I reviewed the current head ac1681ac and found the following correctness issues and test gaps that should be addressed before merge.

Correctness issues

[P1] DELETE can silently succeed without deleting rows when ignore-delete=true

PaimonDmlCommandUtils.checkDelete() accepts every DEDUPLICATE table unconditionally, without checking CoreOptions.IGNORE_DELETE.

In the Paimon 1.3.1 dependency used by this branch, DeduplicateMergeFunction deliberately discards retract records when ignore-delete=true. This PR emits RowKind.DELETE, so both a standalone DELETE and a MERGE DELETE branch can commit successfully while leaving the existing row unchanged.

Please reject DELETE/MERGE DELETE during analysis when delete records are configured to be ignored, or otherwise provide semantics that guarantee the row is removed.

[P1] UPDATE cannot set a nullable column to NULL on merge-engine=partial-update

checkUpdate() explicitly permits PARTIAL_UPDATE, and the new write path sends a complete UPDATE_AFTER row containing the explicit NULL value.

However, Paimon's partial-update merge function only updates non-null fields. As a result:

UPDATE t SET nullable_col = NULL WHERE id = 1;

can report success while retaining the old value. The same issue applies to a MERGE UPDATE assignment.

Please either reject SQL UPDATE/MERGE UPDATE for partial-update tables until explicit NULL can be represented correctly, or add a connector-specific encoding/write path that distinguishes "field not supplied" from "set field to NULL".

[P2] Schema-change retry reuses the old immutable write target

The UPDATE/DELETE/MERGE commands create and pin a PaimonWriteTarget before constructing InsertIntoTableCommand. InsertIntoTableCommand retries by replanning the same already-bound LogicalPaimonTableSink; it does not reload the Paimon table or rebuild the write target.

The new CollectRelation handling can help detect a target schema change, but it does not refresh the pinned columns/table serialized to the writer. A concurrent remote schema change can therefore cause repeated retry failures or a mismatch between the analyzed output schema and the writer table generation.

Please rebuild the row-level DML plan/write target on retry, and add a concurrent schema-change test.

[P2] MERGE does not detect multiple source rows matching one target row

The implementation directly joins source and target and emits one changelog record per joined row. There is no cardinality check for multiple source rows matching the same target primary key. For deduplicate tables this can produce last-write-wins behavior whose result depends on execution/write order, instead of rejecting an ambiguous MERGE.

Please define and enforce the expected cardinality semantics and add a duplicate-source-key test.

Existing regression/coverage failures

• external_table_p0/paimon/test_paimon_write_boundary still asserts that UPDATE, DELETE, and MERGE are unsupported. It now fails in External Regression and must be updated to validate the new behavior.
• BE incremental coverage is currently 0/2 for the changed C++ lines.
• FE incremental coverage is only 9/378 (2.38%).
• The new regression verifies only a default deduplicate table with fixed buckets plus one UPDATE, one DELETE, and one mixed MERGE.

Missing test scenarios

Please add coverage for at least:

1. ignore-delete=true: standalone DELETE and MERGE DELETE must not silently succeed.
2. merge-engine=partial-update:
   • update a non-null value;
   • explicitly set a nullable column to NULL;
   • MERGE UPDATE with an explicit NULL;
   • delete behavior with partial-update.remove-record-on-delete;
   • sequence-group removal options.
3. merge-engine=aggregate with and without aggregation.remove-record-on-delete.
4. MERGE with duplicate source keys matching one target key.
5. All unsupported-engine negative paths for UPDATE, DELETE, and MERGE, not only UPDATE on first-row.
6. Schema changes between initial analysis, retry, writer binding, and commit.
7. Multiple conditional MATCHED/NOT MATCHED clauses and branch-priority behavior.
8. Dynamic bucket modes, sequence fields, partitioned tables, and reordered/mixed-case columns.
9. UPDATE/DELETE with CTEs, joins/USING, aliases, ORDER BY/LIMIT where supported, and zero matched rows.
10. Failure atomicity: writer/commit failure after mixed INSERT/UPDATE/DELETE records must not partially apply the MERGE.
11. Large MERGE/UPDATE/DELETE workloads covering join spill, Paimon writer spill/backpressure, memory limits, cancellation, and OOM protection.
12. JNI/Thrift compatibility for the changed Java open signature and the new CHANGELOG write mode, including a clear mixed-version/rolling-upgrade behavior.

Given the two silent data-correctness failures above and the directly related red regression, I do not think this is ready to merge yet.

[suxiaogang223]

run buildall

[hello-stephen]

FE UT Coverage Report

Increment line coverage 3.49% (17/487) 🎉
Increment coverage report
Complete coverage report

[suxiaogang223]

run buildall

[Gabriel39]

I found two remaining row-level DML correctness gaps on the current head. Both can let a statement succeed while Paimon silently chooses or retains a different row version.

[suxiaogang223]

run buildall

[Gabriel39]

I completed a follow-up review of the current head ef5b271d. The latest commit addresses the previous inline findings about modifying sequence.field and duplicate primary keys among the rows emitted by the same NOT MATCHED INSERT branch. I found the following remaining issues.

[P1] rowkind.field overrides the RowKind emitted by Doris

PaimonWriteSchema.tableRow() sets the InternalRow RowKind from the Doris operation column, but Paimon 1.4.2 uses RowKindGenerator.getRowKind() when rowkind.field is configured. That generator takes precedence over row.getRowKind().

PaimonRowChangeCapabilities currently does not reject or otherwise neutralize this table option. Consequently:

• DELETE can be converted back into INSERT/UPDATE when the stored rowkind field contains +I or +U, so the statement succeeds without deleting the row.
• A MERGE NOT MATCHED INSERT carrying -D in that field can become a physical DELETE.
• Updating the rowkind field can turn an UPDATE into a DELETE or another operation.

Please either reject row-level UPDATE/DELETE/MERGE on tables configured with rowkind.field, or construct a pinned writer table where the generator cannot override the operation selected by Doris. Please also add UPDATE, DELETE, and MERGE coverage for this option.

[P1] NOT MATCHED INSERT does not detect a primary key that already exists in the target

checkInsertPrimaryKeyUniqueness() only checks for duplicate projected keys among the INSERT rows emitted by this MERGE statement. It does not check those keys against existing target rows.

For example, given an existing target row (id=1, status='old'), a source row (id=1, status='new'), and:

ON t.id = s.id AND t.status = s.status

the source row is classified as NOT MATCHED and can emit an INSERT with id=1. A Paimon deduplicate table does not raise a primary-key violation; it applies upsert/sequence semantics and can silently replace or retain a row even though the SQL MERGE branch classified it as an insert.

Please either validate projected INSERT keys against the target primary keys, or restrict the MERGE ON condition so that it guarantees complete primary-key matching. A regression test should cover an additional non-key ON predicate producing this collision.

[P1] Nondeterministic INSERT key expressions are evaluated separately for validation and writing

The uniqueness projection calls generateFinalExpression() to materialize INSERT keys, and the final sink projection calls it again for the actual output row. Expressions such as rand() or uuid() can therefore produce one key during validation and a different key during writing. The uniqueness assertion can pass while the actual sink rows collide.

Please materialize the selected operation and row values once, then use the same slots for both uniqueness validation and the final sink output.

[P2] Unmatched rows form one unbounded NULL window partition

generateTargetMatchCount() partitions by target primary-key slots before branch filtering. In a LEFT JOIN, every NOT MATCHED row has NULL target-key slots, so all unmatched source rows enter the same window partition whenever the MERGE also contains a MATCHED clause.

A normal mixed MERGE with a small number of updates and millions of inserts can therefore create a single shuffle/buffer hotspot and cause excessive spill, memory growth, or OOM. Please run the target cardinality check only for matched rows, without placing all unmatched rows into one NULL partition, and add a mostly-unmatched large-input test.

Additional planner performance concern

Analyzer.buildAnalyzerJobs() now runs a second full bottom-up BindExpression pass for every analyzed statement, although the new expressions are introduced only by sink binding. The applied-aware conditions avoid most transformations, but every plan still pays for another traversal and rule matching. Please scope the rebind to newly created sink nodes if possible, or provide planner benchmark evidence showing that the global pass does not cause a meaningful analysis-time regression.

The current regression suite does not cover the four cases above. Given the silent row-operation/key correctness failures, I do not think this is ready to merge yet.

[hello-stephen]

BE UT Coverage Report

Increment line coverage 50.00% (4/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	58.38% (24785/42451)
Line Coverage	42.69% (250203/586129)
Region Coverage	38.72% (197663/510520)
Branch Coverage	40.15% (90601/225674)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 50.00% (4/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.66% (30477/41374)
Line Coverage	57.65% (335354/581742)
Region Coverage	54.44% (278657/511883)
Branch Coverage	55.45% (124996/225436)

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 1.85% (10/542) 🎉
Increment coverage report
Complete coverage report