benchmarks/history: re-measure the series 1-up and add v0.9.0

[iainmcgin]

Adds v0.9.0 to the performance history — but not by appending a column, because the README forbade that and the reason turned out to understate the problem.

Why the whole series was re-measured

The stated reason: every committed run predates -align-loops=64. A v0.9.0 column measured with the current flags would sit beside rows measured without them, and the README's standing warning says to re-measure the series when adding the next release.

The reason found on the way: measurement was 32-way self-concurrent — 32 copies of a shape's binary at once, one per physical core, for 32 samples in one wall-clock pass. Measuring the same v0.8.0 binaries one at a time on a pinned core, box otherwise idle:

• faster on 41 of 42 benchmarks, median +5.9%
• worst case +44.7% (media_frame/json_encode — largest dataset, most allocation-heavy, exactly where contention should bite)
• spread improved despite dropping 32 samples to 3: p50 3.59% → 0.76%, p90 8.31% → 4.93%

That is not a constant offset that cancels out in cross-release comparison. It is a per-shape bias, so it distorts comparisons between shapes, which is what this matrix exists to support.

The series is therefore measured 1-up, with parallelism coming from several machines rather than several cores of one.

The backport

mesh and column_batch are measured back across every release. The harness belongs to the tooling rather than to any release, so shapes added late can still be measured retroactively — and it materially changes what v0.9.0's decode work looks like:

	v0.8.0 → v0.9.0
packed_tile/decode (short arrays — the shape that existed at the time)	+32.3%
column_batch/decode (long columns — what the work was actually for)	+113.3%
column_batch/merge	+124.3%
mesh/merge	+154.9%

Judged only on the shapes we happened to be tracking when it landed, that work would have been understated roughly threefold.

Each historical-benchmark/vX.Y.Z branch now carries the backported harness (pushed separately), so the new cells remain rebuildable — the design's reproducibility claim depends on it.

A regression, recorded rather than buried

Two cells moved the wrong way and are not noise:

benchmark	v0.7.1	v0.8.0	v0.9.0	spread
google_message1_proto3/decode_view	998.2	911.0	836.3	2.37%
media_frame/decode_view	53324	60188	55252	3.76%

Both -8.2% on spreads well under that. google_message1_proto3 has now declined two releases running, which clears the ±5% band the docs use to separate signal from noise. These are the two most message-dense shapes; the leading hypothesis is #250 (inline singular message fields), and owned decode for both stays inside the band (-1.8%, -2.4%) rather than falling with the view path, so it looks view-specific. Not root-caused — annotated so it isn't rediscovered from a chart later.

run-series.sh

New, and deliberately thin: it deals binaries across boxes and calls bench-on-metal.sh per binary rather than reimplementing the remote path. It retries provisioning across AZs (metal spot capacity was exhausted across three us-west-2 AZs during this run; it completed in us-east-2) and records instance ids so a crashed dispatcher still leaves a cleanup list.

Verification

240 captures across 80 binaries (10 releases × 8 shapes × 3 runs), zero failures. Every number quoted above was computed from the committed run files before being written down — including one claim I had to correct, having written "flat-to-up" for a decode pair that is actually -1.8%/-2.4%.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}