Skip to content

feat(table): drop dangling equality deletes during RewriteDataFiles#947

Merged
zeroshade merged 2 commits into
apache:mainfrom
laskoviymishka:feat/drop-dangling-eq-deletes
Apr 30, 2026
Merged

feat(table): drop dangling equality deletes during RewriteDataFiles#947
zeroshade merged 2 commits into
apache:mainfrom
laskoviymishka:feat/drop-dangling-eq-deletes

Conversation

@laskoviymishka
Copy link
Copy Markdown
Contributor

Closes #946.

Sustained CDC workloads (Postgres → Iceberg replication, ~one snapshot per few seconds with one eq-delete per UPDATE/DELETE) accumulate equality-delete files in every snapshot's manifests forever. After compaction, every preexisting eq-delete is provably dead — no surviving applicable data file has seq < eq-delete.seq — but the old RewriteDataFiles only removed position deletes, leaving eq-deletes in place. Manifest fanout grew linearly with the CDC duration.

This change adds three pieces:

  1. table/compaction.DecideDeadEqualityDeletes — a pure predicate that exactly matches the v2 reader (table/scanner.go matchEqualityDeletesToData): E applies to D iff E.seq > D.seq AND (either side has empty partition OR partition tuples match). SpecID is intentionally NOT in the predicate — using it would silently drop eq-deletes the reader still applies under partition-spec evolution.

  2. table/compaction.CollectDeadEqualityDeletes — walks the snapshot's manifests via the existing public Snapshot.Manifests + FetchEntries APIs, builds a SurvivorSurvey, and returns the dead set. Two-pass design: skip the data-manifest walk if there are no eq-delete candidates.

  3. table.RewriteDataFilesOptions.ExtraDeleteFilesToRemove — a new field on the options struct that the executor passes through to ReplaceFiles. The caller (cmd/iceberg/compact.go or any library user) computes dead eq-deletes via CollectDeadEqualityDeletes against the same snapshot and threads them in. The executor itself is policy-free.

The CLI gains a --preserve-dead-equality-deletes flag (default off, i.e., cleanup is on by default — recommended for sustained CDC).

The RewriteDataFiles signature collapses (partialProgress, snapshotProps, opts...) into a single RewriteDataFilesOptions struct. RewriteResult splits RemovedDeleteFiles into RemovedPositionDeleteFiles + RemovedEqualityDeleteFiles for observability.

Tests:

  • table/compaction/eq_delete_decision_test.go pins the predicate against the scanner's rule across mixed-partition / cross-spec / boundary cases.
  • table/compaction/cdc_stress_test.go runs 80 CDC cycles (81 data files + 80 eq-deletes pre-compaction → 1 data file + 0 eq-deletes post). Skipped under -short.
  • table/rewrite_data_files_test.go covers full-rewrite drops, partial-rewrite preserves, partial-progress preserves (per-group cleanup is a follow-up), and the CLI opt-out path.

Closes apache#946.

Sustained CDC workloads (Postgres → Iceberg replication, ~one snapshot
per few seconds with one eq-delete per UPDATE/DELETE) accumulate
equality-delete files in every snapshot's manifests forever. After
compaction, every preexisting eq-delete is provably dead — no
surviving applicable data file has seq < eq-delete.seq — but the old
RewriteDataFiles only removed position deletes, leaving eq-deletes
in place. Manifest fanout grew linearly with the CDC duration.

This change adds three pieces:

1. table/compaction.DecideDeadEqualityDeletes — a pure predicate that
   exactly matches the v2 reader (table/scanner.go
   matchEqualityDeletesToData): E applies to D iff E.seq > D.seq AND
   (either side has empty partition OR partition tuples match). SpecID
   is intentionally NOT in the predicate — using it would silently
   drop eq-deletes the reader still applies under partition-spec
   evolution.

2. table/compaction.CollectDeadEqualityDeletes — walks the snapshot's
   manifests via the existing public Snapshot.Manifests + FetchEntries
   APIs, builds a SurvivorSurvey, and returns the dead set. Two-pass
   design: skip the data-manifest walk if there are no eq-delete
   candidates.

3. table.RewriteDataFilesOptions.ExtraDeleteFilesToRemove — a new
   field on the options struct that the executor passes through to
   ReplaceFiles. The caller (cmd/iceberg/compact.go or any library
   user) computes dead eq-deletes via CollectDeadEqualityDeletes
   against the same snapshot and threads them in. The executor itself
   is policy-free.

The CLI gains a --preserve-dead-equality-deletes flag (default off,
i.e., cleanup is on by default — recommended for sustained CDC).

The RewriteDataFiles signature collapses (partialProgress, snapshotProps,
opts...) into a single RewriteDataFilesOptions struct. RewriteResult
splits RemovedDeleteFiles into RemovedPositionDeleteFiles +
RemovedEqualityDeleteFiles for observability.

Tests:
- table/compaction/eq_delete_decision_test.go pins the predicate
  against the scanner's rule across mixed-partition / cross-spec /
  boundary cases.
- table/compaction/cdc_stress_test.go runs 80 CDC cycles (81 data
  files + 80 eq-deletes pre-compaction → 1 data file + 0 eq-deletes
  post). Skipped under -short.
- table/rewrite_data_files_test.go covers full-rewrite drops,
  partial-rewrite preserves, partial-progress preserves (per-group
  cleanup is a follow-up), and the CLI opt-out path.
@laskoviymishka laskoviymishka marked this pull request as ready for review April 28, 2026 12:34
Comment thread table/compaction/eq_delete_collect.go
Comment thread table/compaction/eq_delete_collect.go
Comment thread table/compaction/eq_delete_decision.go Outdated
Comment thread table/compaction/eq_delete_decision.go Outdated
Comment thread table/compaction/eq_delete_decision.go Outdated
Comment thread table/rewrite_data_files.go Outdated
- eq_delete_collect.go: drop manual EntryStatusDELETED filter, pass
  discardDeleted=true to FetchEntries
- eq_delete_decision.go: use min builtin in AddSurvivor and
  applicableMinSeq; consolidate nil/empty guards
- rewrite_data_files.go: bracket-link doc references so pkg.go.dev
  cross-links them
@zeroshade zeroshade merged commit ead5282 into apache:main Apr 30, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(table): drop dangling equality delete files during RewriteDataFiles

2 participants