Adopt std.benchmark across the tins

[winding-lines]

Every tin hand-rolls its benchmarks: perf_counter_ns around a loop, best-of-N, divide by hand for MB/s. That was fine when there was one tin. It is now thirteen, the numbers are quoted on magmalake.org and in every README, and the method has known weaknesses. Mojo ships std.benchmark, which fixes most of them.

What the hand-rolled harness gets wrong

Nothing stops the optimiser eliding the work. hashes, snappy and roaring time pure functions whose results are often discarded. We currently avoid dead-code elimination by accident — a stray print or an accumulator that happens to escape — not by construction. std.benchmark.compiler.keep exists precisely for this:

def keep[T: AnyType, origin: Origin, //](ref[origin] value: T)

There is also std.benchmark.memory.clobber_memory() for the store-elision case.

Iteration counts are guesses. Each bench picks a fixed N. run() chooses batches adaptively:

benchmarking continues until min_runtime_secs has elapsed and either max_runtime_secs or max_iters is achieved

Best-of-N is not a statistic. Several READMEs carry caveats like "wide run-to-run variance on this shared dev machine". Report exposes mean, fastest/slowest batch mean, iteration count and warmup total — enough to report a number with a spread instead of an apology.

Throughput is computed by hand. Every repo divides bytes by seconds itself. BenchMetric has the units built in:

metric	unit
BenchMetric.elements	GElems/s
BenchMetric.bytes	GB/s
BenchMetric.flops	GFLOPS/s

Results are prose, not data. Numbers are pasted into READMEs by hand and go stale — the org README drifted from the site for exactly this reason. Bench writes Format.csv, Format.tabular or Format.table, which a script can diff between commits.

Proposed shape

Two tiers, because the tins are not all the same kind of thing.

Tier 1 — microbenchmarks (hashes, snappy, roaring, thrift, parts of avro): use std.benchmark.run with keep, and ThroughputMeasure(BenchMetric.bytes, n) for the rate. These are tight loops over a buffer, exactly what the module is designed for.

Tier 2 — whole-file / whole-scan (parquet, iceberg, objectstore): the unit of work is milliseconds and involves IO, so adaptive iteration matters less and setup cost dominates. Use Bench + bench_with_input where it fits; keep an explicit timed region where it does not. Do not force tier 1 onto tier 2 — a 4 ms Parquet read does not want 100 warmup iterations against a cold page cache.

Where a number is IO-bound or otherwise not a fair run() target, say so in the bench output rather than quietly reporting a shape the harness cannot measure honestly.

Scope

• Pilot on hashes.mojo — three pure functions, the clearest case, and it proves the pattern including keep (bench: move to the shared bench.mojo harness hashes.mojo#1; stable-only, see the comment below)
• Roll tier 1 across snappy, roaring, thrift, avro
• Assess tier 2 for parquet, iceberg, objectstore — report what Bench can and cannot express for them before converting anything
• Emit Format.csv alongside human output so numbers can be diffed across commits
• Re-measure and update the README and magmalake.org tables from the new harness; expect some published numbers to move, and treat that as the point rather than something to paper over
• Decide whether benches run in CI (guarding against regressions) or stay manual (they are slow, and shared runners are noisy)

Non-goals

Not chasing faster numbers. This is about making the existing ones trustworthy and reproducible. If a number gets worse because keep stopped the optimiser deleting the work, that is the harness finding a real bug in the measurement — publish the corrected figure.

[winding-lines]

Pilot done: magmalake/hashes.mojo#1.

The measurement side worked as hoped. Warmup plus adaptive batching replaced a single cold pass, ThroughputMeasure replaced hand arithmetic, and CSV output came almost free. The published numbers moved up — 1.48 / 1.67 / 1.29 GB/s against ~1.3 / ~1.5 / ~1.2 — because the old harness timed one pass over a buffer it had just built. Spread over five runs is now within 2.7%, tight enough to drop the "wide run-to-run variance" hedge the README carried.

One finding changes the plan for the rest of this issue. std.benchmark is not source-compatible between stable 1.0.0 and current nightly, and there is no call form that compiles on both. Bencher.iter has lost its parameter form on nightly; its remaining value form rejects a @parameter closure, and a plain closure cannot infer a capture convention on either toolchain. Net effect: nightly std.benchmark cannot express a benchmark that reads data it did not build inside the timed region. run() and bench_function split the same way.

Every tin here gates on stable and nightly, so each conversion means a stable-only bench task. In hashes I moved it to [feature.stable.tasks] so pixi run bench fails as an unknown task rather than as a wall of overload notes.

That is a genuine cost, but I don't think it should stop the rollout — benches are developer tools, not gates, and better measurement is worth running them on one toolchain. Worth filing upstream either way; I'll add the link here once it is in.

• Pilot on hashes.mojo

[winding-lines]

Change of approach on the pilot, after looking at how marrow does this.

marrow does not use std.benchmark's runner at all — it takes Bencher as a timing primitive and owns discovery, calibration, repetitions and output. Doing the same removes the stable-only blocker I reported above entirely, so the harness now lives in a new shared tin, magmalake/bench.mojo, rather than being reinvented per repo. I verified every primitive it needs (@parameter closures capturing locals, a parameter-form iter on our own struct, __functions_in_module() discovery, rebind into a parameter position) on stable 1.0.0 and nightly before writing it.

What that changes for this issue:

• The two-tier proposal collapses to one tier. bench.mojo handles both shapes; tier 2 just uses fewer repetitions and a longer min_runtime_secs.
• The stable-only cost is much smaller than I said. The bench env is still stable, but only because magmalake tins build their .mojopkg with mojo-compiler 1.0.0 and nightly cannot load it — an org-wide packaging constraint that has nothing to do with benchmarking. Vendoring the source with -I ../bench.mojo/src works on both.
• CSV becomes JSON. --out PATH writes an array carrying every per-repetition timing plus mean/min/max/median/stddev — strictly more than Format.csv gave, and the right input for trend reporting.
• Saving and reporting is now its own piece of work. marrow's shape is worth copying closely: per-commit envelope, a rolling data.json capped at 200 runs, a static dashboard, and a regression rule of max(5%, 2×CV) against the previous window so noisy benchmarks do not cry wolf. Worth a separate issue.

magmalake/hashes.mojo#1 is updated to this approach and ready to review.

The pilot also found a real bug in the measurement, which was the point. xxh64 is 28% faster when it is the only xxh64 call in a module (40 ms per 64 MiB against 51) because the compiler specialises it into a lone caller. A benchmark calling each hash once measures a form no real consumer gets. Filed as magmalake/hashes.mojo#2; the bench file now anchors a second call site deliberately so the published figure is the shared one.

Revised checklist for the remaining tins — the rollout is cheaper than I estimated, since each repo now adds a dependency and a bench feature rather than its own harness.

[winding-lines]

Decision: reports live per repo, on each repo's own gh-pages, the way marrow does it. Each tin keeps its own history next to its own code; nothing has to be aggregated into magmalake.org, and a repo can be handed to someone else without taking its benchmark history hostage.

Propagation status

Seven PRs open, all CI-green on stable and nightly × macOS and Linux:

repo	PR	shape
hashes.mojo	#1	3 benchmarks
snappy.mojo	#1	4 — compress/decompress × compressible/random
roaring.mojo	#1	3 — add / serialize / deserialize
thrift.mojo	#1	6 — footer read/write × large/small, skip, varints
lz4.mojo	#1	4 — block/frame × compress/decompress
avro.mojo	#1	20 — datum, OCF, deflate, and the Value/cursor matrix
zstd.mojo	#1	14 — one per (shape, level, direction)

What the conversions turned up

Four measurement bugs, which was the point of the exercise:

1. roaring's add benchmark had the RNG inside the timed loop. Roughly three quarters of the reported 5–11 s was random_ui64, not Bitmap32.add. It is 1.30 s once the values are generated in setup.
2. hashes' xxh64 is 28% faster when it has exactly one call site (xxh64 is 28% slower when it has more than one call site hashes.mojo#2) — a benchmark calling it once measures a specialisation no real consumer gets.
3. Cold-pass inflation across every fast operation. lz4 compress_block read ~9.9 GB/s and is 25.3; zstd compressible level 1 read ~10.2 and is 20.5. At 2–5 ms per pass, building and first-touching a 64 MiB buffer was most of what got timed.
4. best-of-N flattered avro by a few percent. Its numbers now sit slightly lower; fastavro re-measured in the same session moved by the same amount, so the ratios hold.

Also worth recording: several conversions needed an explicit keep after b.iter for values the closure captured but the body never mentioned again — Mojo destroys at last use and a capture does not count. It surfaces as a crash inside the library under test, not as a lifetime error. Documented in bench.mojo's README.

Two harness gaps found and fixed while doing this: a max_iters ceiling (a near-free body needed 10⁸ iterations to fill a second, so calibration alone ran 45 s), and SI-prefix scaling in the rate column (873 thousand column chunks a second was rendering as 0.0009 GElems/s).

Tier 2: assessed, not converted

• parquet.mojo — blocked on an API question, not on the harness. ParquetReader owns its List[UInt8], so a re-entrant timed body needs a fresh copy per iteration, and that memcpy lands inside the timed region. On a 5 MB fixture that is a 5–15% inflation of every decode number. The fix is a borrowing constructor (ParquetReader.from_span), which is a library change and does not belong in a bench PR. Worth its own issue.
• iceberg.mojo — bench/run_bench.sh generates a 15 MB warehouse and builds a PyIceberg venv before anything runs, and compares against PyIceberg afterwards. The Mojo half converts cleanly; the orchestration around it does not want to move into the harness. Probably: convert the timed scans, leave the script as the thing that sets up and compares.
• objectstore.mojo — poor fit. It needs MinIO and a local HTTP server, the work is IO-bound, and the harness re-enters a body once per phase, which would re-upload fixtures repeatedly. Calibrating iteration counts against a network is measuring the network. Leave it hand-rolled unless someone wants a specific number tracked.

Remaining on this issue: the saving and reporting layer — per-commit JSON, a rolling data.json on each repo's gh-pages, a static dashboard, and the max(5%, 2×CV) regression rule. --out already emits the input it needs.