[Spark] Merge Into: When Not Matched By Source - #2517
Merged
Conversation
JingsongLi
reviewed
Dec 18, 2023
JingsongLi
left a comment
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
Before this PR, can we create an abstraction for MergeIntoTableTest. Just like TestTrinoITCase, every spark module has a class to extend this, test merge into feature for every Spark versions.
Contributor
kerwin-zk
added a commit
to kerwin-zk/paimon
that referenced
this pull request
Aug 10, 2026
…ctions
`MergeIntoPaimonTable` extracts the target-only conjuncts of the merge
condition and uses them to prune the target table before the full outer
join (`filteredTargetPlan` / `targetOnlyCondition`).
That pruning is sound for `WHEN MATCHED` and `WHEN NOT MATCHED`: a target
row that fails a target-only conjunct can never satisfy the whole merge
condition, so it can never be matched, and dropping it cannot change the
outcome of those actions.
It is not sound for `WHEN NOT MATCHED BY SOURCE`. The pruned-away rows are
exactly the population that clause is defined over, so their actions are
silently skipped -- no error, no warning, just fewer rows changed.
For example, with a partitioned table and
MERGE INTO target t USING source s
ON t.a = s.a AND t.pt = 'p1'
WHEN MATCHED THEN UPDATE SET t.b = s.b
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET t.c = 'stale'
every row outside `pt = 'p1'` should be updated to `stale` (no source row
can match it), but none of them is.
The pruning was introduced together with MERGE INTO itself in apache#2331, one
month before `WHEN NOT MATCHED BY SOURCE` was added in apache#2517, and its
safety argument was never revisited.
Note that the V2 row-level paths (`ReplaceData` / `WriteDelta`) are rewritten
by Spark and are not affected, so the same statement currently produces
different results depending on whether the table qualifies for
`SparkTable.supportsV2RowLevelOps`. Primary key tables never qualify, so they
always take the affected V1 path.
This disables the pruning when the merge has any `WHEN NOT MATCHED BY SOURCE`
action. Setting `targetOnlyCondition` to `None` covers all three places it
feeds: `filteredTargetPlan`, `findCandidateDataSplits` and
`targetDSWithFilePathCol`.
A follow-up can restore part of the pruning by handling the excluded rows as
a separate not-matched-by-source-only stream, which avoids joining them
against the source while still applying their actions.
kerwin-zk
added a commit
to kerwin-zk/paimon
that referenced
this pull request
Aug 10, 2026
…ctions
`MergeIntoPaimonTable` extracts the target-only conjuncts of the merge
condition and uses them to prune the target table before the full outer
join (`filteredTargetPlan` / `targetOnlyCondition`).
That pruning is sound for `WHEN MATCHED` and `WHEN NOT MATCHED`: a target
row that fails a target-only conjunct can never satisfy the whole merge
condition, so it can never be matched, and dropping it cannot change the
outcome of those actions.
It is not sound for `WHEN NOT MATCHED BY SOURCE`. The pruned-away rows are
exactly the population that clause is defined over, so their actions are
silently skipped -- no error, no warning, just fewer rows changed.
For example, with a partitioned table and
MERGE INTO target t USING source s
ON t.a = s.a AND t.pt = 'p1'
WHEN MATCHED THEN UPDATE SET t.b = s.b
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET t.c = 'stale'
every row outside `pt = 'p1'` should be updated to `stale` (no source row
can match it), but none of them is.
The pruning was introduced together with MERGE INTO itself in apache#2331, one
month before `WHEN NOT MATCHED BY SOURCE` was added in apache#2517, and its
safety argument was never revisited.
Note that the V2 row-level paths (`ReplaceData` / `WriteDelta`) are rewritten
by Spark and are not affected, so the same statement currently produces
different results depending on whether the table qualifies for
`SparkTable.supportsV2RowLevelOps`. Primary key tables never qualify, so they
always take the affected V1 path.
This disables the pruning when the merge has any `WHEN NOT MATCHED BY SOURCE`
action. Setting `targetOnlyCondition` to `None` covers all three places it
feeds: `filteredTargetPlan`, `findCandidateDataSplits` and
`targetDSWithFilePathCol`.
A follow-up can restore part of the pruning by handling the excluded rows as
a separate not-matched-by-source-only stream, which avoids joining them
against the source while still applying their actions.
kerwin-zk
added a commit
to kerwin-zk/paimon
that referenced
this pull request
Aug 11, 2026
…ctions
`MergeIntoPaimonTable` extracts the target-only conjuncts of the merge
condition and uses them to prune the target table before the full outer
join (`filteredTargetPlan` / `targetOnlyCondition`).
That pruning is sound for `WHEN MATCHED` and `WHEN NOT MATCHED`: a target
row that fails a target-only conjunct can never satisfy the whole merge
condition, so it can never be matched, and dropping it cannot change the
outcome of those actions.
It is not sound for `WHEN NOT MATCHED BY SOURCE`. The pruned-away rows are
exactly the population that clause is defined over, so their actions are
silently skipped -- no error, no warning, just fewer rows changed.
For example, with a partitioned table and
MERGE INTO target t USING source s
ON t.a = s.a AND t.pt = 'p1'
WHEN MATCHED THEN UPDATE SET t.b = s.b
WHEN NOT MATCHED BY SOURCE THEN UPDATE SET t.c = 'stale'
every row outside `pt = 'p1'` should be updated to `stale` (no source row
can match it), but none of them is.
The pruning was introduced together with MERGE INTO itself in apache#2331, one
month before `WHEN NOT MATCHED BY SOURCE` was added in apache#2517, and its
safety argument was never revisited.
Note that the V2 row-level paths (`ReplaceData` / `WriteDelta`) are rewritten
by Spark and are not affected, so the same statement currently produces
different results depending on whether the table qualifies for
`SparkTable.supportsV2RowLevelOps`. Primary key tables never qualify, so they
always take the affected V1 path.
This disables the pruning when the merge has any `WHEN NOT MATCHED BY SOURCE`
action. Setting `targetOnlyCondition` to `None` covers all three places it
feeds: `filteredTargetPlan`, `findCandidateDataSplits` and
`targetDSWithFilePathCol`.
A follow-up can restore part of the pruning by handling the excluded rows as
a separate not-matched-by-source-only stream, which avoids joining them
against the source while still applying their actions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #xxx
Tests
API and Format
Documentation