Cascade Utf8 dict codes so they bitpack (#303)#305
Merged
Conversation
encodeUtf8Cascade exposed the dict values pool as an open child (#299 Finding 2b) but left the codes as a raw vortex.primitive leaf — paying the full U8/U16/U32 code width per row. The Rust reference bitpacks dict codes (fastlanes.bitpacked), and the primitive dict path here already exposes its codes the same way; only the Utf8 path was missing it. Now both children are left open: codes (index 0) as a typed primitive array the cascade bitpacks, and the values pool (index 1) for the FSST/VarBin competition. No wire/reader change — the reader already decodes bitpacked dict codes (it reads Rust files that use them). nyc-311 re-encode: 1934.23 MB -> 1862.23 MB (1.10x -> 1.06x vortex-jni); the win spans every per-chunk Utf8 dict column, not only the street columns that surfaced it. Composes with #302 (they compress different children). Round-trip + 217 Java-writes-Rust-reads interop tests pass; full verify green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 25, 2026
Closed
dfa1
added a commit
that referenced
this pull request
Jul 25, 2026
) On a wide, high-cardinality file the per-column code buffers (~37 MB each, ADR 0021) sum past the 1 GB budget, and eviction demotes the largest-retained — i.e. the highest-cardinality — columns to per-chunk dictionaries, repeating their values pool every chunk. nyc-311 (~30 admitted string columns ≈ 1.15 GB retained) lost its three Cross Street / Street Name columns this way. Raise the default to 2 GB, which fits that file with headroom (measured: 2 GB and 3 GB both match the 8 GB result, so retained tops out ~1.15 GB) while still bounding the pathological many-wide-columns risk. Constrained-heap writers can still lower it via withGlobalDictMaxRetainedBytes(...). nyc-311 re-encode: 1934.23 MB -> 1879.34 MB (1.10x -> 1.07x vortex-jni). This overlaps #305 (both improve the street columns, via keeping-global vs bitpacked codes): stacked on #305 the budget adds ~35 MB net (1862 -> 1827 MB). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The per-chunk Utf8 dictionary path emitted its codes as a raw
vortex.primitiveleaf (full U8/U16/U32 width per row), while the Rust reference bitpacks them. This was the largest remaining per-column gap after #299 (the street columns in #303).Encoding tree for a street column (Cross Street 1), before vs after:
How
encodeUtf8Cascadeexposed the values pool as an open child (#299 Finding 2b) but left codes as a raw leaf. Now both children are open: codes (index 0) as a typed primitive array the cascade bitpacks, and the values pool (index 1) for the FSST/VarBin competition. The primitive dict path already did this; only the Utf8 path was missing it.No wire/reader change — the reader already decodes bitpacked dict codes (it reads Rust files that use them; conformance + interop suites cover it).
Impact (real nyc-311 corpus, off
main)−72 MB — larger than the ~54 MB street-column estimate because it helps every per-chunk Utf8 dict column. Composes with #302 (that fix bitpacks the FSST offset/length children — different columns); together they put vortex-java well under the reference.
Testing
encodeCascadenow exposes two open children (codes + values)../mvnw verifygreen.Addresses #303 (finding #1 — bitpacked codes). Finding #2 (per-chunk vs global dict fallback for these columns) remains open there as separate upside.
🤖 Generated with Claude Code