The timing loop, the repetition model and the interval

[tamnd]

Closes #5.

This is the measurement half of bench-core. No benchmark lives in the crate, only the loop, the statistics and the repetition model, so that the timing code can be read in one sitting by somebody who wants to check it.

The repetition model

Kalibera and Jones is usually remembered as run it more times. The actual contribution is that variance lives at a particular level, and the repetition budget should be spent at the level that carries it. A Pilot is a small balanced design across builds, processes and iterations. Pilot::decompose splits the variance across those three levels using the ordinary nested random effects arithmetic, and plan gives the budget to the dominant level and two repetitions to each of the others.

Two and not one for the settled levels, because one gives no way to notice later that a level started carrying variance after all, which happens the first time somebody changes a compiler flag.

The decomposition is written as plain nested loops rather than iterator chains. It is arithmetic a reader should be able to check against a textbook, and a chain of folds is harder to check than it is to write.

The statistics

The headline is the median with a percentile bootstrap confidence interval. The mean is carried because everyone else reports it, not because it is the better statistic. The minimum gets its own column rather than being folded into anything, because the minimum is a statistic about the luckiest run and printing it where a reader expects a typical value is the most common distortion in this field.

The bootstrap seed is configuration rather than something taken from the clock, so summarising the same samples twice gives the same interval. That matters more than it sounds, because the stopping rule is keyed on the width of that interval and a rule built on a number that moves on its own would stop somewhere different every run.

Two thousand resamples rather than the ten thousand people reach for out of habit. Ten thousand is right for a percentile of a tail. For an interval on a median it moves the endpoints by less than the measurement noise they describe, and the stopping rule pays this cost repeatedly while a run is in progress.

The stopping rule

Stop::should_stop takes the series being measured and nothing else. It cannot see a baseline, a rival, or yesterday's number, because the signature does not let it. Stop is non_exhaustive and no variant carries anything but a scalar, so nobody outside the crate can add a variant that sees a second series.

That is the whole design. Adding repetitions until a comparison becomes significant is how a result gets manufactured, and it does not feel like cheating while you are doing it, because you are just collecting more data and more data is good. The defence is not discipline, it is not having the number available at the point where the decision is made.

The issue asks for a test that a comparison driven rule is not reachable. The unreachable half is a type signature, so the test checks the observable half: the same workload measured on a day when the rival was far behind and on a day when it was a hair ahead stops at the same sample and produces the same summary.

Timing::check_every exists because a bootstrap over the whole series is not free and one more sample rarely moves the interval enough to change the answer. A run ends at a multiple of it, which is worth knowing when reading a sample count back, so there is a test that says so.

What is here

• stats.rs, the summary, the bootstrap, and a small deterministic generator so the crate does not take a dependency for eight lines of arithmetic
• repetition.rs, the pilot, the variance decomposition and the budget
• measure.rs, the loop, the stopping rule and time
• 24 tests across three files

Unused dependencies were dropped from bench-core while I was in there. The crate now takes serde and thiserror and nothing else.