feat(delete-vector): [1/N] decode deletion-vector-v1 puffin blobs#2866
Open
mbutrovich wants to merge 3 commits into
Open
feat(delete-vector): [1/N] decode deletion-vector-v1 puffin blobs#2866mbutrovich wants to merge 3 commits into
mbutrovich wants to merge 3 commits into
Conversation
Add DeleteVector::deserialize to parse a deletion-vector-v1 blob ([length][magic][roaring][crc32], portable 64-bit roaring) into a DeleteVector. Verifies the length prefix, CRC-32, and magic before decoding, and leaves cardinality validation to the caller that holds the delete file's record_count. Tests cover round-trips (including a run-optimized bitmap that exercises the RUN-container decode path) and the length, magic, and CRC error paths. Refs apache#2792.
anoopj
approved these changes
Jul 21, 2026
Collaborator
Author
This was referenced Jul 22, 2026
hsiang-c
approved these changes
Jul 22, 2026
hsiang-c
left a comment
Contributor
There was a problem hiding this comment.
Thanks @mbutrovich, LGTM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
deletion-vector-v1Puffin blobs) #2792.What changes are included in this PR?
Adds
DeleteVector::deserialize, which decodes adeletion-vector-v1Puffin blob into aDeleteVector(aRoaringTreemapof deleted row positions). This is the first, self-contained piece of deletion-vector read support (epic #2792): it adds no scan, reader, or planning changes and can be reviewed in isolation.The blob layout follows the Iceberg Puffin spec and matches Iceberg-Java's
BitmapPositionDeleteIndex/RoaringPositionBitmap:lengthcounts the magic and vector bytes (not itself or the CRC).crc32fast) covers the magic and vector.vectoris a roaring bitmap in the portable 64-bit format, read directly byRoaringTreemap::deserialize_from(roaring 0.11.x implements this official format).Validation order is length prefix, then CRC (before decoding any payload), then magic, then the roaring decode, so a corrupt blob yields a single clear error rather than an opaque roaring failure. Cardinality is intentionally not checked here; the caller validates the decoded length against the delete file's
record_countonce the manifest metadata is available (a later PR in the epic).Also adds
crc32fastas a dependency.This overlaps with #2414, which implemented the same decode. That PR has been idle since review feedback, so this is submitted in its place to keep the epic moving; thanks to @Shekharrajak for the original contribution and for informing this implementation.
Are these changes tested?
Yes, unit tests in
delete_vector.rs:RoaringTreemap::optimize) so the RUN-container decode path is exercised, since Iceberg-Java run-length-encodes deletion vectors before writing.This decode path is also exercised end to end against deletion vectors written by Spark / Iceberg-Java in a draft DataFusion Comet PR (apache/datafusion-comet#4887), which reads real V3 merge-on-read tables through this code and verifies the deleted rows match Spark. That gives cross-implementation confidence that it decodes what Iceberg actually writes, not just what these tests encode.