Refactor/module layout by gargiulofrancesco · Pull Request #20 · spiraldb/onpair

gargiulofrancesco · 2026-07-06T15:31:30Z

Summary

Breaking refactor of the public API ahead of the 0.1.0 release, plus new
compressed-domain search capabilities.

Breaking changes

Split the crate into modules: column, core, encoding, decoding, and search.
Column/ColumnView are the high-level owned entry points; embedders that
manage their own buffers use the core/decoding/search APIs directly.
Sealed dictionaries behind a trust/validation boundary.
Renamed the training dictionary-width knob to MaxDictBits /
Config::max_dict_bits, making explicit that it is a dictionary-size
budget; runtime code width is derived from dictionary size via
CompactDictionary::code_bits.

Added

Compressed-domain equality, prefix, and substring search APIs.
Column::into_raw, CompactDictionary::into_raw, and
code_bits_for_num_tokens for embedders that store OnPair buffers in
their own layout.
Safe buffer-checked try_decode_into alongside the zero-alloc unsafe
buffer decode path.

Performance

Into-buffer over-copy decode for random access.
Decode fast/exact tail split by output bytes instead of a fixed code count.
Faster equality search via length-check + early-out loop.
Unchecked wide dictionary build on the trusted view.

Release

Bumps the crate version to 0.1.0 and finalizes the changelog; once this
merges, publishing the drafted GitHub release as v0.1.0 will push it to
crates.io via the publish workflow.

Reorganize the flat src layout into layered modules: core (dictionary, types, offset), encoding (parser, trainer, lpm, config), decoding (kernels + scalar copy + fat table), column, and search. Decode now writes exactly-sized output buffers: the fast path 16-byte over-copies all but the final MAX_TOKEN_SIZE tokens, which are copied at their true length, removing the DECOMPRESS_BUFFER_PADDING requirement. FatTable moves to decoding/fat.rs, gains decode_to_vec, and drops its unused +16 table row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add a Dictionary (owned) / DictionaryView (borrowed) trait pair over the two physical token-vocabulary layouts. Rename the compact structs Dictionary/DictionaryView -> CompactDictionary/CompactDictionaryView; relocate FatTable to WideDictionary plus a borrowed WideDictionaryView. Reorganize the dictionary into a core/dictionary/ folder (traits in mod.rs, compact.rs, wide.rs). Make decode_into/decode_to_vec/decoded_len generic over DictionaryView so one kernel serves both layouts; add to_wide/to_compact conversions and ColumnView::wide_dict. Per-token reads (token, token_len) move onto DictionaryView; the owned dictionary keeps storage, lifecycle, and metadata only. Move code_bits_for to core::types beside its inverse max_dict_size (const fn returning BitWidth); rename decoding::scalar -> decoding::copy; documentation cleanups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…count The fast 16-byte over-store needs >= MAX_TOKEN_SIZE output bytes ahead of it. The exact-copy tail is now the trailing tokens spanning the last < MAX_TOKEN_SIZE bytes (found by walking back from the end summing token lengths), instead of a fixed last-MAX_TOKEN_SIZE-codes tail. For multi-byte tokens a fixed code-count tail forced a short row's entire output through the slow exact path; measuring bytes keeps the leading tokens on the fast path — a win for fine-grained random access of short strings. Safety is unchanged: every fast-path token still has >= MAX_TOKEN_SIZE output bytes remaining. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Mirror CompactDictionary's structured `# Invariants` section in the wide module: move the inline invariant list from the struct doc up to the module level, keep the shared logical invariants (sorted, complete, unique) verbatim, and adapt the representation-specific sizing clause. The wide form has no read-padding invariant; its safety rationale stays in the token_ptr SAFETY comment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…e_unchecked Move decoded_len from a DictionaryView method to a free function in the decoding module, and add num_tokens plus the unchecked byte_unchecked accessor that the sorted-dictionary search hot loop relies on. Also standardizes a # Safety heading in decoding/copy.rs and fixes the CompactDictionaryView::to_wide intra-doc links in wide.rs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Answer equality, prefix, and substring queries directly over the code stream without decoding rows, exploiting the dictionary's sorted/complete/unique invariants. Each query is prepared once (tokenize / PrefixQuery / ContainsTable) and applied per row by a free function over a &[Token]; ColumnView gains rows_equal_to, rows_starting_with, and rows_containing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add decompress_into / decompress_row_into, which decode the whole column or a single row into a caller-provided [MaybeUninit<u8>] buffer with no allocation (one buffer reusable across rows), plus row_decoded_len to size it. Marked unsafe for now — the caller upholds the buffer-length precondition rather than the methods asserting it — so the safe-vs-unsafe overhead can be benchmarked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Collapse the nested decrement (if threshold > 2 { threshold - 1 } else { 2 }) into a single guarded `threshold -= 1`, only when ratio < 0.5 and threshold > 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Apply cargo fmt to the files whose only change is formatting. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Replace `codes == query` (which lowers to a `memcmp` call) with an explicit length check followed by a short-circuiting zip loop over tokens. This drops the call overhead and returns on the first differing token, speeding up the equality scan — especially on the common mismatch case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Establish an explicit trust boundary for dictionary buffers and unify how malformed compressed data is surfaced. - Seal the trusted types: private fields on CompactDictionary/WideDictionary and their views, constructed only via crate-internal `from_raw`, plus a sealed `DictionaryView` trait. A trusted view is now validated by construction, so its unchecked token accessors are sound. - Add UntrustedDictionary/UntrustedDictionaryView for raw deserialized buffers, crossing into the trusted forms via `validate` (checked) or `trust_unchecked` (the unsafe backdoor). - Add core::validate: `InvalidColumn` (safety + conformance violations) and `panic_malformed`; add `ColumnView::validate` as a recoverable pre-flight. - Bounds-check every code in `decode_into`/`decoded_len` (panic, never UB) and drop the old unsafe "codes in range" caller contract. - Move bulk decode to a caller-owned buffer sized `decoded_len + DECODE_PADDING`; remove the Vec-returning `decompress`/`decode_to_vec` and the dual copy16/exact-tail path (now a single copy16 per token). - Make `pad_for_decoder` the free `pad_raw`; the trainer accumulates raw buffers and seals once. Drop wide->compact conversion and `byte_unchecked`. - Update benches, examples, and the lib doctest to the new decode API and the `bytes()`/`offsets()` accessors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CompactDictionaryView::to_wide ran its copy loop with bounds-checked indexing and copy_from_slice — ~5 checks/branches per token that the trusted-view invariants already rule out. Switch to get_unchecked plus a fixed-width copy_nonoverlapping (one 16-byte move per token), exactly as the method's doc already described ("never validates ... valid by construction"). Sound on the safety invariants alone (strictly-increasing offsets, length <= MAX_TOKEN_SIZE, read-padding), which every CompactDictionaryView upholds via validate()/trust_unchecked(); conformance is not required here. Stays sound under the planned trust-model work, where the safety-only path mints a WideDictionary rather than a CompactDictionaryView. Verified: clippy (deny warnings), 120 tests, miri on core::dictionary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Fold the dictionary trust boundary onto the trusted compact types as constructors, instead of a separate untrusted type: CompactDictionary[View]::validate(bytes, offsets) -> Result<_, InvalidColumn> CompactDictionary[View]::new_unchecked(bytes, offsets) (unsafe) Removes UntrustedDictionary / UntrustedDictionaryView and the trusted/untrusted asymmetry. The boundary itself is unchanged: the trusted types' fields stay private, so a value is still only obtainable via the trainer, `validate` (checked), `unsafe new_unchecked`, or `as_view`. `validate_compact` moves into compact.rs; `from_raw` stays the internal pub(crate) mint both doors wrap. Also wraps an over-long line in `to_wide` that cargo fmt flagged — the feature branch doesn't trigger CI, so it had gone unformatted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Replace the Vec-returning `ColumnView::decompress_row` with `decompress_row_into(k, out)`, the per-row analog of `decompress_into`: the same fixed 16-byte over-copy per token, decoded directly over the compact dictionary (no O(num_tokens) wide-table build for a single short row), into a caller-owned buffer sized `row_decoded_len(k) + DECODE_PADDING`. The removed method allocated a fresh `Vec` (geometric reallocs) and did a per-token variable-length copy — the one decode path that ignored the crate's branchless over-copy strategy. The new one is symmetric with the bulk path and lets callers reuse a buffer across rows. Its safety rests on the same trusted, read-padded `CompactDictionaryView` invariants the bulk decode relies on, so it stays panic-not-UB on a malformed column. Test oracles that wanted an owned row now wrap `decompress_row_into` in a local helper per test module (matching the existing `compress_rows` convention); the lib doc example uses the new method. Verified: cargo fmt, clippy (deny warnings, --all-targets), 119 tests, doctest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add `into_raw(self) -> (Vec<u8>, Vec<u32>)`, the inverse of `validate` / `new_unchecked`: it consumes the trusted dictionary and moves its owned buffers out without copying, for handing the read-padded token bytes and the N+1 offsets to another owner (e.g. serializing into an embedder's format) instead of the copy that `bytes()` / `offsets()` would force. Consuming `self` dismantles only the trust wrapper — the buffers are untouched and still conformant (read-padding intact), so they rebuild into a trusted dictionary via `validate` / `new_unchecked`. Verified: cargo fmt, clippy (deny warnings, --all-targets), 120 tests, doctest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add `try_decode_into`, the safe counterpart of the unsafe `decode_into`, for decoding code streams whose output buffer is sized from an external figure (e.g. an embedder's stored per-row lengths) rather than this column's own `decoded_len`. Bounds-checking codes is necessary but not sufficient there: in-range codes can address MAX_TOKEN_SIZE-byte tokens, so a buffer sized from anything but the true decoded length can still overflow. The decode derives its bound from `out` itself, with no per-token output check and no `decoded_len` pre-pass: each round issues `(out.len() - written) / MAX_TOKEN_SIZE` fixed 16-byte over-copies — the largest count provably within `out` — so the hot loop matches `decode_into`; the trailing sub-16-byte stretch (or an entire small decode) is finished with exact per-token copies, so `out` needs no DECODE_PADDING slack (a buffer of exactly `decoded_len` suffices). - Restore `copy::copy_token_bytes` (the exact-length companion to `copy16`, overlapping power-of-two stores, no `memcpy` call) for that tail — it also carries small random-access decodes (`scalar_at`) that never enter the over-copy fast path. - `OutputTooSmall` is the recoverable buffer-capacity outcome, deliberately separate from `InvalidColumn` (a too-small caller buffer is not a column malformation). An out-of-range code stays a panic, consistent with `decode_into` / `decoded_len` and recoverable up front via `ColumnView::validate`. Verified: cargo fmt --check, clippy (deny warnings, --all-targets), 125 tests, doctest, and miri on core::dictionary + decoding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CLAassistant · 2026-07-06T15:31:37Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

codspeed-hq · 2026-07-06T15:37:10Z

Merging this PR will improve performance by 52.27%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 24 improved benchmarks
✅ 8 untouched benchmarks
⏩ 2 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	WallTime	`decompress_all[12]`	885.4 µs	363.3 µs	×2.4
⚡	WallTime	`decompress_all[("p_name", 12)]`	1,381.6 µs	599.1 µs	×2.3
⚡	WallTime	`decompress_all[16]`	885.6 µs	385.7 µs	×2.3
⚡	WallTime	`decompress_all[("p_name", 16)]`	1,184.7 µs	545.5 µs	×2.2
⚡	WallTime	`decompress_all[("l_comment", 12)]`	18.7 ms	9.6 ms	+94.67%
⚡	WallTime	`decompress_all[("o_comment", 12)]`	16.9 ms	8.7 ms	+93.43%
⚡	WallTime	`decompress_all[("o_comment", 16)]`	17.2 ms	10.3 ms	+67.39%
⚡	WallTime	`decompress_all[("l_comment", 16)]`	18.7 ms	11.3 ms	+64.91%
⚡	Simulation	`decompress_all[("l_comment", 12)]`	128.8 ms	82.7 ms	+55.83%
⚡	Simulation	`decompress_all[("o_comment", 12)]`	115.5 ms	75.6 ms	+52.87%
⚡	Simulation	`decompress_all[("l_comment", 16)]`	125.5 ms	82.5 ms	+52.22%
⚡	Simulation	`decompress_all[("o_comment", 16)]`	116.2 ms	77.2 ms	+50.58%
⚡	Simulation	`decompress_all[("p_name", 12)]`	11.3 ms	7.6 ms	+47.83%
⚡	Simulation	`decompress_all[12]`	7 ms	4.8 ms	+45.99%
⚡	Simulation	`decompress_all[16]`	7.2 ms	5 ms	+44.95%
⚡	Simulation	`decompress_all[("p_name", 16)]`	10.7 ms	7.5 ms	+41.85%
⚡	WallTime	`train_and_compress[("p_name", 16)]`	41 ms	33.3 ms	+22.81%
⚡	WallTime	`train_and_compress[("o_comment", 16)]`	474.4 ms	398.7 ms	+19.01%
⚡	WallTime	`train_and_compress[("l_comment", 16)]`	540 ms	454.8 ms	+18.74%
⚡	WallTime	`train_and_compress[12]`	31 ms	26.4 ms	+17.34%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing refactor/module-layout (33ecd70) with develop (3f6a8fb)}

2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

gargiulofrancesco and others added 20 commits June 18, 2026 13:38

refactor(trainer): simplify dynamic-threshold decrement

86ae130

Collapse the nested decrement (if threshold > 2 { threshold - 1 } else { 2 }) into a single guarded `threshold -= 1`, only when ratio < 0.5 and threshold > 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

style: cargo fmt

ca2ed2f

Apply cargo fmt to the files whose only change is formatting. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

chore: bump onpair to 0.1.0

2461e67

refactor(config): clarify max dictionary bits

2e5871c

feat(api): expose raw column parts

86f0679

docs(changelog): finalize 0.1.0 release notes

ed40fef

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

gargiulofrancesco added the changelog/break Breaking change to the public API label Jul 6, 2026

docs(decoding): drop redundant explicit link target for InvalidColumn

33ecd70

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

gargiulofrancesco merged commit a31dfd5 into develop Jul 6, 2026
10 of 11 checks passed

gargiulofrancesco deleted the refactor/module-layout branch July 6, 2026 16:31

gargiulofrancesco mentioned this pull request Jul 7, 2026

Port OnPair to onpair 0.1.0; add compressed-domain LIKE, list_contains, and take kernels vortex-data/vortex#8675

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor/module layout#20

Refactor/module layout#20
gargiulofrancesco merged 21 commits into
developfrom
refactor/module-layout

gargiulofrancesco commented Jul 6, 2026

Uh oh!

CLAassistant commented Jul 6, 2026

Uh oh!

codspeed-hq Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

gargiulofrancesco commented Jul 6, 2026

Summary

Breaking changes

Added

Performance

Release

Uh oh!

CLAassistant commented Jul 6, 2026

Uh oh!

codspeed-hq Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 52.27%

Performance Changes

Footnotes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codspeed-hq Bot commented Jul 6, 2026 •

edited

Loading