Skip to content

perf(table): stream projected equality deletes - #1774

Closed
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/equality-delete-streaming
Closed

perf(table): stream projected equality deletes#1774
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/equality-delete-streaming

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Read equality delete files with only the fields used by the delete key.
  • Build encoded delete keys while record batches are being read.
  • Keep the existing field ID and name mapping behavior.
  • Preserve null handling, type encoding, cancellation, and the current worker limit.
  • Add an equivalence test against the old materialized path.

Why

ReadTable loaded the full equality delete file before the keys were built. This also loaded columns that were not part of the equality key. Streaming the projected columns lowers peak memory use and avoids that extra work.

Checks

  • go test ./table -count=1
  • go test -race ./table -run "EqualityDelete|ReadEqualityDeleteFile" -count=1
  • go vet ./table

Benchmark

For a 100k-row equality delete file on an Apple M1 Pro:

  • Streamed projection: 30.6 MB/op, 11.6 ms/op
  • Materialized table: 161.2 MB/op, 25.0 ms/op

@fallintoplace
fallintoplace marked this pull request as ready for review August 12, 2026 08:56
@fallintoplace
fallintoplace marked this pull request as draft August 12, 2026 09:02
@fallintoplace
fallintoplace marked this pull request as ready for review August 12, 2026 10:36
@fallintoplace
fallintoplace marked this pull request as draft August 12, 2026 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant