Benchmark bool run-end decode over fixed corpora - #9020
Open
joseph-isaacs wants to merge 3 commits into
Open
Conversation
`run_end_decode`'s bool cases give every run the same length and derive its value from the run index, then decode that one array over and over. Both properties flatter the decoder: the run length is a loop-invariant constant, and the per-run branch on the run's value -- the branch an adaptive decode strategy turns on -- is perfectly learnable after one pass. What the numbers then measure is partly the branch predictor's memory of one array. Add cases that decode a set of arrays per sample, each array independently generated from a fixed seed, with every run length drawn at random around the target average. No run boundary or run value sequence repeats within a sample, and the working set is too large to sit in L1. Seeds are fixed, so the corpus is byte-identical on every run and in CI. Two shapes bracket what a scan sees -- 128 arrays of 16,384 elements, and 4 of 100,000 -- across average run lengths of 4, 16 and 64, which span shorter than a machine word, around a word, and longer than a word. Each is run with 50/50 values, 90/10 values, and 50/50 values over 90% validity. Benchmarks only; no library code changes. Signed-off-by: "Claude" <noreply@anthropic.com>
The corpus loop black-boxed each decoded array and dropped it on the spot, so every sample measured one deallocation per array alongside the decode. Decode kernels differ in how many buffers they allocate and at what sizes, so that is not a neutral constant folded into every arm -- it is a term that moves with the thing being measured. Collect into a vector instead and return it, so divan drops it after the timer stops. The vector is allocated with capacity by `with_inputs`, which also runs outside the timed region, leaving a push per array as the only addition -- a store and a length bump. Signed-off-by: "Claude" <noreply@anthropic.com>
Run lengths were drawn uniformly from `1..=2*avg - 1`. That gives the right mean, but a flat band with a hard cap: for an average of 64, no run in the case ever exceeds 127, and every length in the band is equally likely. The spread of run lengths, not the mean, is what sets how often a run crosses a 64-bit word boundary, so the shape of this distribution is not incidental to what the benchmark measures. Draw from a normal about the mean instead, with a standard deviation of a third of it, which puts the clamp at one element three standard deviations out so the truncated tail is negligible and the realised mean stays close to the nominal one. Signed-off-by: "Claude" <noreply@anthropic.com>
joseph-isaacs
force-pushed
the
claude/runend-bool-decode-benches
branch
from
July 28, 2026 11:19
d3ac4e5 to
bd674b8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
run_end_decode's bool cases give every run the same length and derive its value from the run index, then decode that one array over and over. Both properties flatter a decoder:What those numbers partly measure is the branch predictor's memory of one array, not the decoder. Real bool columns are neither uniform in run length nor repeated.
This lands the measurement first, on its own, so a following change to the decode kernel shows up in CI as a delta against a benchmark that was already there.
What changes are included in this PR?
A new bench target,
run_end_decode_bool_corpus:Two shapes bracket what a scan sees:
128x16k4x100kcrossed with average run lengths of 4, 16 and 64 — shorter than a machine word, around a word, longer than a word — and three value mixes: 50/50 values, 90/10 values, and 50/50 values over 90% validity. 18 cases, reported with an
ItemsCountcounter.Local walltime for the whole target is a few seconds; the largest single case is ~6.7 ms per sample.
Existing bench targets are untouched, so their CI baselines are unaffected.
One known limitation
Adjacent runs may carry the same value. For a canonically encoded bool column that would be wrong — equal neighbours would merge, forcing strict alternation — but bool run-end arrays are not produced that way. They arise as the validity of an integer run-end array, reusing its ends, or as a comparison pushed into one. Neither merges on bool equality, so non-alternating and skewed values are the realistic shape and the 90/10 cases are meaningful.
Checks run:
cargo build -p vortex-runend --benches,cargo bench -p vortex-runend --bench run_end_decode_bool_corpus,cargo clippy -p vortex-runend --all-targets --all-features,cargo +nightly fmt --all,git diff --check.What APIs are changed? Are there any user-facing changes?
None. Benchmarks only, no library code touched. Adds
rand_distrto this crate's dev-dependencies; it is already a workspace dependency used byvortex-array's benches.