Skip to content

Cascade Utf8 dict codes so they bitpack (#303)#305

Merged
dfa1 merged 1 commit into
mainfrom
dict-codes-cascade
Jul 25, 2026
Merged

Cascade Utf8 dict codes so they bitpack (#303)#305
dfa1 merged 1 commit into
mainfrom
dict-codes-cascade

Conversation

@dfa1

@dfa1 dfa1 commented Jul 25, 2026

Copy link
Copy Markdown
Owner

What

The per-chunk Utf8 dictionary path emitted its codes as a raw vortex.primitive leaf (full U8/U16/U32 width per row), while the Rust reference bitpacks them. This was the largest remaining per-column gap after #299 (the street columns in #303).

Encoding tree for a street column (Cross Street 1), before vs after:

vortex-jni                     vortex-java before             vortex-java after
  DICT                           per-chunk vortex.dict          per-chunk vortex.dict
    codes: fastlanes.bitpacked     codes: vortex.primitive  ❌     codes: fastlanes.bitpacked  ✅
    values: vortex.fsst            values: vortex.fsst            values: vortex.fsst

How

encodeUtf8Cascade exposed the values pool as an open child (#299 Finding 2b) but left codes as a raw leaf. Now both children are open: codes (index 0) as a typed primitive array the cascade bitpacks, and the values pool (index 1) for the FSST/VarBin competition. The primitive dict path already did this; only the Utf8 path was missing it.

No wire/reader change — the reader already decodes bitpacked dict codes (it reads Rust files that use them; conformance + interop suites cover it).

Impact (real nyc-311 corpus, off main)

vortex-java vs vortex-jni (1762 MB)
Before (main) 1934.23 MB 1.10×
After 1862.23 MB 1.06×

−72 MB — larger than the ~54 MB street-column estimate because it helps every per-chunk Utf8 dict column. Composes with #302 (that fix bitpacks the FSST offset/length children — different columns); together they put vortex-java well under the reference.

Testing

  • Updated unit test: encodeCascade now exposes two open children (codes + values).
  • Dict round-trip + 217 Java-writes-Rust-reads interop tests pass.
  • Full ./mvnw verify green.

Addresses #303 (finding #1 — bitpacked codes). Finding #2 (per-chunk vs global dict fallback for these columns) remains open there as separate upside.

🤖 Generated with Claude Code

encodeUtf8Cascade exposed the dict values pool as an open child (#299 Finding 2b)
but left the codes as a raw vortex.primitive leaf — paying the full U8/U16/U32
code width per row. The Rust reference bitpacks dict codes (fastlanes.bitpacked),
and the primitive dict path here already exposes its codes the same way; only the
Utf8 path was missing it.

Now both children are left open: codes (index 0) as a typed primitive array the
cascade bitpacks, and the values pool (index 1) for the FSST/VarBin competition.
No wire/reader change — the reader already decodes bitpacked dict codes (it reads
Rust files that use them).

nyc-311 re-encode: 1934.23 MB -> 1862.23 MB (1.10x -> 1.06x vortex-jni); the win
spans every per-chunk Utf8 dict column, not only the street columns that surfaced
it. Composes with #302 (they compress different children). Round-trip + 217
Java-writes-Rust-reads interop tests pass; full verify green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dfa1
dfa1 merged commit 8ab484c into main Jul 25, 2026
6 checks passed
@dfa1
dfa1 deleted the dict-codes-cascade branch July 25, 2026 06:18
dfa1 added a commit that referenced this pull request Jul 25, 2026
)

On a wide, high-cardinality file the per-column code buffers (~37 MB each,
ADR 0021) sum past the 1 GB budget, and eviction demotes the largest-retained —
i.e. the highest-cardinality — columns to per-chunk dictionaries, repeating their
values pool every chunk. nyc-311 (~30 admitted string columns ≈ 1.15 GB retained)
lost its three Cross Street / Street Name columns this way.

Raise the default to 2 GB, which fits that file with headroom (measured: 2 GB and
3 GB both match the 8 GB result, so retained tops out ~1.15 GB) while still bounding
the pathological many-wide-columns risk. Constrained-heap writers can still lower it
via withGlobalDictMaxRetainedBytes(...).

nyc-311 re-encode: 1934.23 MB -> 1879.34 MB (1.10x -> 1.07x vortex-jni). This
overlaps #305 (both improve the street columns, via keeping-global vs bitpacked
codes): stacked on #305 the budget adds ~35 MB net (1862 -> 1827 MB).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant