Skip to content

fix(deletion_vector): read Java bitmap64 deletion vectors - #785

Open
jackylee-ch wants to merge 2 commits into
apache:mainfrom
jackylee-ch:fix/deletion-vector-bitmap64
Open

fix(deletion_vector): read Java bitmap64 deletion vectors#785
jackylee-ch wants to merge 2 commits into
apache:mainfrom
jackylee-ch:fix/deletion-vector-bitmap64

Conversation

@jackylee-ch

@jackylee-ch jackylee-ch commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

A deletion vector written by Java in the 64-bit format (Bitmap64DeletionVector)
cannot be read: read_from_bytes accepts only the v1 magic, so every scan touching
such a file fails with Invalid magic number, naming the v1 magic alone.
Java's DeletionVector.read dispatches on both.

Two things differ beyond the value. The v2 magic is written little-endian, and
DeletionFile.length() counts the length prefix and the CRC for v2 but neither for
v1. The second also breaks the read range: the factory asked for length + 8, past
the end of a v2 entry, so a vector ending its index file was rejected by the storage
layer before parsing.

Fix: detect either magic, apply each format's length convention, and decode the
payload's outer layer here rather than through RoaringTreemap::deserialize_from
that decoder validates each inner roaring32 bitmap but inserts bucket keys straight
into a map, so a duplicate key silently replaces a bucket and its deletes vanish.
Java rejects that in OptimizedRoaringBitmap64.readBitmapCount / readKey, and the
same checks are applied: count range, key bounds, strictly ascending. A duplicate
key, a descending key, a bogus count or a truncated payload now error instead of
decoding partially. Java run-length encodes before writing and emits every bucket
key densely, so tests cover both shapes.

Read side only; the writer still emits v1. Positions above u32::MAX are rejected
rather than truncated — every API here is already roaring32-bound, and reaching one
needs a data file with over 4.29e9 rows. A negative bitmapLength read from the
file now errors instead of wrapping the size guard into passing.

/// deletes; a data file with more than `u32::MAX` rows would be needed to
/// reach it.
fn from_bitmap64_bytes(bitmap_data: &[u8]) -> crate::Result<Self> {
let treemap = roaring::RoaringTreemap::deserialize_from(bitmap_data).map_err(|e| {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Reject duplicate or out-of-order bitmap64 bucket keys before they can drop deletes

roaring::RoaringTreemap::deserialize_from validates each inner roaring32 bitmap, but its outer decoder simply inserts each u32 key into a BTreeMap; it does not enforce Java OptimizedRoaringBitmap64.deserialize's requirement that keys are strictly increasing. A duplicate key therefore overwrites the earlier bucket silently. If an index is corrupted so two buckets carry key 0, deleted positions from the first bucket reappear as live rows instead of the vector being rejected.

I verified this with a two-bucket payload containing key 0 / bitmap {1} followed by key 0 / bitmap {2}: this method succeeds and returns only {2}; the equivalent Java reader rejects the second key. Please decode/validate the outer count and keys explicitly (including nonnegative/range/order checks) before assembling the treemap, and add duplicate plus descending-key cases.

@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Confirmed. Outer layer is decoded here now, mirroring readBitmapCount/readKey: count range, key bounds, strictly ascending. Duplicate or descending keys error instead of dropping deletes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants