fix(encoding): handle u16 num_levels overflow in miniblock List encoding - #6989
Merged
Conversation
The miniblock structural codec stores each chunk's num_levels as a u16 in the v2.2 on-disk header. For shapes where a single chunk packs more than 65 535 levels (e.g. the HNSW persistence pattern: a dense level-0 prefix followed by ~6x as many mostly-empty higher-level rows, where the final chunk via slice_rest absorbs every trailing empty), num_chunk_levels as u16 silently truncated and the decoder short-read the chunk, corrupting the round trip (struct.rs:382 length-mismatch panic on read). repdef_too_sparse_for_miniblock previously only looked at the global levels/values ratio, which is healthy for this shape (~1.19) so the heuristic fell through to miniblock and the corruption fired. Extend the heuristic with any_chunk_levels_overflow_u16, a single linear pass that simulates the encoder's chunking (MAX_MINIBLOCK_VALUES visible values per non-last chunk; final chunk via slice_rest absorbs trailing empties) and returns true if any chunk would carry more than u16::MAX levels. Early-exit when the rep/def levels are absent (no list nesting) or when the total def-level count itself fits in u16, so dense / non-list paths pay zero added cost (encoder bench delta vs baseline: <0.5%, within noise). As a codec safety net, replace the silent num_chunk_levels as u16 cast at the chunk-build site with a u16::try_from that errors with a clear 'fullzip required' message — dormant in normal operation now that the heuristic catches the shape upstream, but prevents future shapes from silently corrupting data if they sneak past the heuristic. v2.2 is a stable on-disk format (is_unstable() = self >= Next, and V2_2 < Next), so widening the header isn't an option; the proper fix at the codec layer is to route affected shapes away from miniblock.
…rride Address cosmetic review items from the round-0 cross reviews: - Correct '5x' doc comment to '~6x' (matches the 240k/40k test ratio). - Rename local 'chunk_val_size' to 'max_visibles_per_chunk' for clarity. - Use Error::invalid_input_source instead of Error::internal at the chunk-levels try_from site — this is a user-shape error, not an internal invariant violation. - Emit a log::warn when the user explicitly requested STRUCTURAL_ENCODING_MINIBLOCK but the heuristic vetoed it and routed to fullzip, so the override is observable rather than silent. No behavioural change for already-supported shapes; only metadata/log output and the error variant differ.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
touch-of-grey
changed the title
The plural 'visibles' is flagged by crate-ci/typos as a misspelling of 'visible'. Use the singular adjective form for the local variable names in any_chunk_levels_overflow_u16.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Xuanwo
added a commit
that referenced
this pull request
May 29, 2026
Resolve conflicts with #6989: drop the competing repdef_too_sparse_for_miniblock / any_chunk_levels_overflow_u16 heuristic in favor of this PR's structural page splitting, which keeps the dense prefix on mini-block pages instead of falling back to fullzip. Port #6989's HNSW regression tests with assertions updated to expect the split mini-block layout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Surfaced while benchmarking mem_wal HNSW flush at scale (#6901 made flush
rebuild the persisted secondary indexes; the in-memory graph's
__neighbors/__distscolumns areList<UInt32>/List<Float32>).Flushing a generation larger than ~32k nodes at v2.1 panicked the
lance-cputhread withchunk_bytes <= max_chunk_size; writing thoseindex files at v2.2 cleared the write panic but produced a corrupt read,
panicking the struct decoder with
Mismatch in length (at offset=83300) expected 100 got 92(counts varyrun-to-run with HNSW non-determinism).
HNSW::schema()is shared withevery IVF_HNSW index, so this is a general codec correctness bug, not
specific to mem_wal.
Root cause: the miniblock structural codec stores each chunk's
num_levelsasu16in the v2.2 on-disk header(
encodings/logical/primitive.rs:467). The HNSW persistence shape — adense level-0 prefix followed by ~6× as many empty higher-level rows —
has a healthy global levels/values ratio (~1.19), so
repdef_too_sparse_for_miniblockfell through to miniblock. The finalvalue chunk via
slice_restthen absorbed all trailing empties (242 048levels for a 40k-dense + 240k-empty case),
num_chunk_levels as u16silently truncated to 45 440, and the decoder short-read.
v2.2 is a stable on-disk format (
V2_2 < Next), so the per-chunk headercannot widen. Fix is code-only:
repdef_too_sparse_for_miniblockwithany_chunk_levels_overflow_u16, a single linear pass over the deflevels that mirrors the encoder's chunking (
MAX_MINIBLOCK_VALUESvisible values per non-last chunk;
slice_restfor the final) andreturns true if any chunk would carry more than
u16::MAXlevels.Routes affected shapes to fullzip through the existing dispatcher.
Early-exits at
max_visible_level = Noneanddef_levels.len() <= u16::MAXkeep dense / small-page paths zero-cost (encoder bench deltawithin criterion noise).
as u16becomesu16::try_from(...).map_err(Error::invalid_input_source)?so anyfuture shape that slips past the heuristic surfaces a clear error
rather than corrupting data.
A
log::warn!records when the heuristic overrides an explicitSTRUCTURAL_ENCODING_MINIBLOCKrequest. Newtest_list_hnsw_shape_auto_routes_around_miniblock_overflow_v2_2reproduces the
Mismatch in lengthpanic onHEAD~1and round-tripscleanly on this commit; existing
test_sparse_large_string_listparametrization still passes both encodings.
Follow-ups (not in this PR): (a) a real value chunk straddling the
simulation's 4096-value boundary can split an empty cluster across two
sim chunks; the codec safety net catches that as a clean error rather
than corruption — tightening to a sliding window is follow-up. (b)
List<Dictionary<...>>on the too-sparse path hits a pre-existingunreachable!inencode_full_zip; not introduced here but theheuristic broadens the surface.