Phase B: per-column dictionary / delta / zigzag-varint encodings - #13
Merged
Conversation
The columnar segment format already reserved the Encoding discriminants (Dictionary, Delta, Zigzag) when it shipped, so this is pure writer/ reader work — no format version bump, segments written before this change still load unchanged via the Plain path. Per-column choice: event_id Plain (high cardinality, ~unique per row) kind Plain (3 values; Plain+zstd good enough) correction_ref Plain (sparse + structured) account_id Dictionary (heavy repeat) subscription_id Dictionary (heavy repeat, nullable) product_id Dictionary meter_id Dictionary model_id Dictionary (nullable) source Dictionary unit Dictionary timestamp_ms Delta (near-monotonic; tiny deltas) quantity Zigzag (small +/- values pack to 1-2 bytes) dimensions Plain (BTreeMap; bincode + zstd handles it) ingested_at_ms Delta Dictionary payload: `(Vec<String>, Vec<u32>)` — unique values + per-row index. For nullable strings, `Vec<Option<u32>>` so None is preserved without a sentinel. Delta payload: `Vec<i64>` of running differences; reader does prefix- sum to reconstruct. Zigzag-varint payload: `u32 LE count` + concatenated zigzag→varint bytes. The count is needed because varints are variable-width. Zigzag maps signed → unsigned (`-1 → 1`, `1 → 2`, …) so small absolute values stay small unsigned. Reader is permissive: it accepts Plain OR the appropriate alternative for each column type. This means a future writer change (e.g., adding RLE for `kind`) doesn't break compatibility — segments with mixed encoding history coexist. Impact: the existing `compression_reduces_size_on_repetitive_data` test still passes its <500KB bar, but the new `dictionary_encoding_shrinks_repetitive_id_columns` test tightens the bar to <250KB. Real workloads with mostly-static account/product/ meter columns should see comparable or larger wins. Five new tests in tests/encodings.rs: - dictionary_encoding_shrinks_repetitive_id_columns - dictionary_round_trip_with_many_distinct_values (incl. Option nulls) - delta_encoding_handles_out_of_order_timestamps (incl. negatives + MAX) - zigzag_varint_round_trips_edge_cases (incl. i128::MIN/MAX) - zigzag_varint_packs_small_quantities_tight Total tests: 82 (was 77; +5). Clean under RUSTFLAGS=-D warnings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The columnar segment format reserved
Encoding::{Dictionary, Delta, Zigzag}discriminants when it shipped — this PR implements them. Pure writer/reader work; no format version bump.Per-column choices
event_idkindcorrection_refaccount_id/product_id/meter_id/source/unitsubscription_id/model_idtimestamp_ms/ingested_at_msquantitydimensionsPayload formats
(Vec<String>, Vec<u32>)(orVec<Option<u32>>for nullable). Unique values plus a per-row index.Vec<i64>of running differences; reader does prefix-sum.u32 LE count+ concatenated zigzag→varint bytes. The count is needed because varints are variable-width.Compatibility
The reader is permissive: for each column type, it accepts
PlainOR the appropriate alternative. This means segments written before this PR still load unchanged (everything wasPlain), and a future encoder change for any column (e.g., adding RLE forkind) doesn't break old segments.Impact
The existing
compression_reduces_size_on_repetitive_datatest (10k events with one constant account) still passes its<500 KBbar. The newdictionary_encoding_shrinks_repetitive_id_columnstest tightens that bar to<250 KBon the same payload. Real workloads with mostly-static IDs should see comparable or larger wins.Tests
Five new tests in
tests/encodings.rs:dictionary_encoding_shrinks_repetitive_id_columns— tightened size bounddictionary_round_trip_with_many_distinct_values— includesOption<String>nullsdelta_encoding_handles_out_of_order_timestamps— out-of-order + negatives +i64::MAX/2zigzag_varint_round_trips_edge_cases—0,±1,±128,i128::MAX,i128::MINzigzag_varint_packs_small_quantities_tight— 1000 small-quantity events under 8 KBTest plan
cargo build --all-targetsclean with-D warningscargo test --all-targets— 82 tests pass (was 77; +5)🤖 Generated with Claude Code