Skip to content

Benchmark bool run-end decode over fixed corpora - #9020

Open
joseph-isaacs wants to merge 3 commits into
developfrom
claude/runend-bool-decode-benches
Open

Benchmark bool run-end decode over fixed corpora#9020
joseph-isaacs wants to merge 3 commits into
developfrom
claude/runend-bool-decode-benches

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

run_end_decode's bool cases give every run the same length and derive its value from the run index, then decode that one array over and over. Both properties flatter a decoder:

  • the run length is a loop-invariant constant, so per-run setup that would cost something on irregular data is hoisted or predicted away;
  • the per-run branch on the run's value — the branch an adaptive decode strategy turns on — is perfectly learnable after one pass over a single fixed array.

What those numbers partly measure is the branch predictor's memory of one array, not the decoder. Real bool columns are neither uniform in run length nor repeated.

This lands the measurement first, on its own, so a following change to the decode kernel shows up in CI as a delta against a benchmark that was already there.

What changes are included in this PR?

A new bench target, run_end_decode_bool_corpus:

  • decode a set of arrays per sample, not one array repeatedly. Each array is independently generated from its own seed, so no run boundary or run value sequence repeats within a sample and the predictor has nothing to memorise.
  • run lengths drawn from a normal distribution about the target average, with a standard deviation of a third of the mean. The spread of run lengths, not the mean, is what sets how often a run crosses a machine word boundary, so the shape matters to what is being measured. Three sigma puts the clamp at one element far enough out that the truncated tail is negligible and the realised mean stays close to the nominal one.
  • fixed seeds, so the corpus is byte-identical on every run and in CI.
  • working sets (2M and 400k elements per sample) too large to sit in L1.
  • decoded arrays are collected and returned, so divan drops them after the timer stops. Dropping each array as it is produced would put a deallocation per array inside the measurement, and decode strategies differ in how many buffers they allocate and at what sizes, so that term moves with the thing being measured rather than being a constant.

Two shapes bracket what a scan sees:

shape arrays elements each per sample
128x16k 128 16,384 2M
4x100k 4 100,000 400k

crossed with average run lengths of 4, 16 and 64 — shorter than a machine word, around a word, longer than a word — and three value mixes: 50/50 values, 90/10 values, and 50/50 values over 90% validity. 18 cases, reported with an ItemsCount counter.

Local walltime for the whole target is a few seconds; the largest single case is ~6.7 ms per sample.

Existing bench targets are untouched, so their CI baselines are unaffected.

One known limitation

Adjacent runs may carry the same value. For a canonically encoded bool column that would be wrong — equal neighbours would merge, forcing strict alternation — but bool run-end arrays are not produced that way. They arise as the validity of an integer run-end array, reusing its ends, or as a comparison pushed into one. Neither merges on bool equality, so non-alternating and skewed values are the realistic shape and the 90/10 cases are meaningful.

Checks run: cargo build -p vortex-runend --benches, cargo bench -p vortex-runend --bench run_end_decode_bool_corpus, cargo clippy -p vortex-runend --all-targets --all-features, cargo +nightly fmt --all, git diff --check.

What APIs are changed? Are there any user-facing changes?

None. Benchmarks only, no library code touched. Adds rand_distr to this crate's dev-dependencies; it is already a workspace dependency used by vortex-array's benches.

`run_end_decode`'s bool cases give every run the same length and derive its
value from the run index, then decode that one array over and over. Both
properties flatter the decoder: the run length is a loop-invariant constant,
and the per-run branch on the run's value -- the branch an adaptive decode
strategy turns on -- is perfectly learnable after one pass. What the numbers
then measure is partly the branch predictor's memory of one array.

Add cases that decode a set of arrays per sample, each array independently
generated from a fixed seed, with every run length drawn at random around the
target average. No run boundary or run value sequence repeats within a sample,
and the working set is too large to sit in L1. Seeds are fixed, so the corpus
is byte-identical on every run and in CI.

Two shapes bracket what a scan sees -- 128 arrays of 16,384 elements, and 4 of
100,000 -- across average run lengths of 4, 16 and 64, which span shorter than
a machine word, around a word, and longer than a word. Each is run with 50/50
values, 90/10 values, and 50/50 values over 90% validity.

Benchmarks only; no library code changes.

Signed-off-by: "Claude" <noreply@anthropic.com>
claude added 2 commits July 28, 2026 09:14
The corpus loop black-boxed each decoded array and dropped it on the spot,
so every sample measured one deallocation per array alongside the decode.
Decode kernels differ in how many buffers they allocate and at what sizes, so
that is not a neutral constant folded into every arm -- it is a term that
moves with the thing being measured.

Collect into a vector instead and return it, so divan drops it after the
timer stops. The vector is allocated with capacity by `with_inputs`, which
also runs outside the timed region, leaving a push per array as the only
addition -- a store and a length bump.

Signed-off-by: "Claude" <noreply@anthropic.com>
Run lengths were drawn uniformly from `1..=2*avg - 1`. That gives the right
mean, but a flat band with a hard cap: for an average of 64, no run in the
case ever exceeds 127, and every length in the band is equally likely. The
spread of run lengths, not the mean, is what sets how often a run crosses a
64-bit word boundary, so the shape of this distribution is not incidental to
what the benchmark measures.

Draw from a normal about the mean instead, with a standard deviation of a
third of it, which puts the clamp at one element three standard deviations
out so the truncated tail is negligible and the realised mean stays close to
the nominal one.

Signed-off-by: "Claude" <noreply@anthropic.com>
@joseph-isaacs
joseph-isaacs force-pushed the claude/runend-bool-decode-benches branch from d3ac4e5 to bd674b8 Compare July 28, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants