Skip to content

[iceberg] Fix compaction commits emitting 'overwrite' instead of 'replace' in Iceberg snapshot operation#7702

Merged
JingsongLi merged 1 commit intoapache:masterfrom
0dunay0:fix/iceberg-compaction-replace-operation
Apr 27, 2026
Merged

[iceberg] Fix compaction commits emitting 'overwrite' instead of 'replace' in Iceberg snapshot operation#7702
JingsongLi merged 1 commit intoapache:masterfrom
0dunay0:fix/iceberg-compaction-replace-operation

Conversation

@0dunay0
Copy link
Copy Markdown
Contributor

@0dunay0 0dunay0 commented Apr 26, 2026

Summary

Fixes #7683.

When Paimon compacts an LSM table with the Iceberg metadata committer enabled, the resulting Iceberg snapshot has its operation field set to "overwrite". According to the Iceberg spec, compaction and file rewrite operations should use "replace" instead. Using "overwrite" makes downstream consumers like Polaris REST catalogs and query engines think the compaction was a data mutation, which is incorrect.

The problem was in IcebergCommitCallback.createWithDeleteManifestFileMetas(), which hardcoded IcebergSnapshotSummary.OVERWRITE whenever files were removed, without checking whether the commit was actually a compaction.

The fix:

  • Add an IcebergSnapshotSummary.REPLACE constant for the "replace" operation
  • Pass Snapshot.CommitKind into createWithDeleteManifestFileMetas() and use REPLACE when commitKind == COMPACT, OVERWRITE otherwise
  • Add an assertion in IcebergCompatibilityTest.testDeleteImpl() to verify that compaction commits produce operation: "replace" in the Iceberg snapshot metadata

@0dunay0
Copy link
Copy Markdown
Contributor Author

0dunay0 commented Apr 26, 2026

@JingsongLi @plusplusjiajia Please have a look at this PR

Copy link
Copy Markdown
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit 4a26749 into apache:master Apr 27, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][Iceberg] IcebergCommitCallback emits "overwrite" for compaction commits; should emit "replace" per Iceberg spec

2 participants