chore(bench): delete host calibration; fixed host, raw numbers, verifiable - #749
Merged
Conversation
Benchmarks move to a fixed reference workstation, which removes GHA host rotation but introduces contention: the machine is a real desktop other people use. A run overlapping a game or a build is not a slow measurement, it is a wrong one, and nothing downstream can tell afterwards. run-bench-stable now aborts when the host is busy, and records busy and governor into apparatus.env so a suspect snapshot is attributable. The signal is non-idle CPU sampled from /proc/stat, deliberately not load average and not login presence. Measured on the reference box: under load, busy read 16.7% while 1-minute load still read 0.09; twelve seconds after the load stopped, busy had cleared to 0.1% while load had climbed to 0.92. Keyed on load, the guard would have admitted the contended run and blocked the clean one. An idle desktop session is not contention, so other logins are reported as context on a trip, never as its cause. Skipped under CI, where the runner is ephemeral and rotation dominates instead. A non-performance governor warns rather than blocks, since the account that runs benches cannot set it without root.
…trim Two changes to how the reference host is measured, both from data taken on it. Governor: run-bench-stable sets performance for the suite and restores the previous value in a finally, via a sudoers rule scoped to one root-owned helper. It no-ops wherever that helper is absent. The benefit is small and was nearly recorded with the wrong sign. Five powersave runs followed by five performance runs showed performance looking much worse -- median spread 6.33% vs 4.20%, worst anchor 26.07% vs 10.88%. Interleaving four paired runs reversed it: median per-anchor CV 1.79% on performance against 1.98% on powersave, median speed delta +0.06%. The sequential comparison was measuring elapsed time, not the governor. Any future apparatus A/B on this host must interleave. The governor is restored rather than left on performance because the box is a desktop somebody else uses. TRIM 1 -> 2: scoring 18 runs as all 306 ordered pairs, where the ideal scale is exactly 1, gives median scale error 1.182% untrimmed, 0.945% at 1, 0.753% at 2, 0.615% at 3, 0.565% at 4. A trim of 1 was too narrow because this host has more than one unstable anchor -- the allocation-heavy ones, whose timing is GC-scheduled. Evidence is single-host, so it shows trimming suppressing GC noise and only infers the host-shape case. Nearly free while BASELINE_ANCHOR_HZ is null; a sub-percent re-ruler afterwards.
BASELINE_ANCHOR_HZ was null since #716, so scale was 1 and nothing was normalized. This captures it, which activates host normalization. Measured 2026-07-28 on the reference host: i9-7980XE 18C/36T, Linux x64, Node v26.0.0, V0_BENCH_TARGET=dist, median of 3, 446 benches across 17 files, all 13 anchors present. Host verified idle at 0.19% busy with the governor held at performance; both are recorded in apparatus.env. These values are now the unit, not a measurement. Re-capturing them re-rulers every stored snapshot, so the docstring spells out the only two legitimate reasons (reference host replaced, anchor suite changed) and that neither is a same-PR change. One caveat found while validating and worth knowing before anyone debugs a confusing scale: anchor throughput depends on suite context. Scored against this baseline, 18 anchors-only runs all landed low, 0.971-0.996, because the baseline was measured with the anchor file running after 400+ other benches and they were not. The pipeline always runs the full suite, so this does not affect it -- but never score an anchors-only run against a full-suite baseline. Expect one spurious whole-panel delta on the first regen after this: the gate diffs a raw pre-apparatus prev against a normalized next. Once only.
johnleider
changed the title
… review Codex and Grok reviewed the apparatus; all six findings reproduced against source or on the reference host. 1. The guard was blind to single-core contention. It sampled only the aggregate /proc/stat line, and one saturated core on a 36-thread host is 2.8% of total CPU -- under any sane whole-machine threshold, while being the likeliest real contention there is. Reproduced: one busy thread read 2.8% and sailed through. Now samples per-CPU too and blocks above 35% on any single one, a limit set from the measured gap between an idle box (0-1% on its busiest core) and a real workload (100%). 2. Governor restore was unsound for any original but powersave. readGovernor can return schedutil, ondemand or mixed(...), which the helper rejects, so the cast to its two-value API failed at runtime and left the desktop pinned to performance -- the exact outcome the comment promised to prevent. The hold now declines to touch an unrestorable governor at all. 3. mkdtempSync sat between setting the governor and the try block, so a throw there leaked performance. Everything fallible now happens first. 4. finally does not run on Ctrl-C. Release is now bound to SIGINT/SIGTERM -- which by itself did nothing, because handlers only run when the event loop turns and execFileSync blocked it for the whole suite. Verified: SIGINT left vitest benching on and the box on performance, the interrupt queued behind the run it was meant to abort. runOnce now spawns async and the handler kills the child; re-verified, the box comes back to powersave. 5. Skipping the guard on CI=true disabled it on the machine it was written for, since the reference host under automation sets CI too. Ephemeral is now read as CI set AND no governor helper present. The host is always inspected regardless, so apparatus.env records conditions behind every artifact even when nothing would be stopped. 6. iowait was counted as idle. A disk-bound neighbour still moves memory bandwidth, evicts page cache and raises interrupt load. Verified: a dd loop read 2.9% aggregate -- invisible -- and 38.4% on one core. apparatus.env gains peak, the busiest-core figure, which is the load-bearing one for attribution.
…on is worth The baseline was captured from a single median-of-3 run, so that run's noise became the unit. It is now the per-anchor median across four independent full-suite runs on the idle reference host. Medianing was necessary but not sufficient, and the measurement that came out of it matters more than the new numbers. Across those four runs, three produced a scale within 0.4% of 1 while the fourth produced 0.963: its allocation-heavy anchors collapsed (object churn -38%, grouped aggregate -27%) while the 446 real benches moved only +1.4%. The median anchor moved -4.6%, so no trim setting rescues it -- the anchors did not track the suite that run. Normalizing by that scale would have injected ~4% of error into every bench rather than removing any. Ruled out as a cause: the async spawn change landed between runs, but a fourth run on the new code behaved normally (scale 0.9957), so the outlier is a property of the anchors and not of the harness. The honest reading is that on an unchanged host this apparatus is a no-op two runs in three and a ~4% distortion in the third. Its value is cross-host -- insurance against a future CPU, kernel or Node major -- and since every artifact stores its raw anchors, that correction stays derivable after the fact. Both the code and the rules doc now say so, and say to read a sub-5% normalized delta as noise.
Member
Author
Second-model review round (Codex GPT-5.5 + Grok 4.5)Six findings, all reproduced against source or on the reference host before being accepted. Fixed in
Re-baseline, and a finding that changes how to read this PR
More important than the new numbers: across those four runs on the same idle host, three produced a scale within 0.4% of 1 while the fourth produced 0.963 — its allocation-heavy anchors collapsed ( The async-spawn change landed between runs and was ruled out as the cause: a fourth run on the new code behaved normally (scale 0.9957). Honest reading: on an unchanged host this apparatus is a no-op two runs in three and a ~4% distortion in the third. Its value is cross-host — insurance against a future CPU/kernel/Node change — and because every artifact stores raw anchors, that correction stays derivable retroactively. A sub-5% normalized delta should be read as noise. Both the code and the rules doc now say this. This also retires the earlier "scale reproducible to 0.012%" claim from the PR body — that run was luckier than it was representative. |
The calibration apparatus corrected for GHA rotating CPUs between runs. That problem was solved by moving to a fixed workstation; what remained was a correction for a variable that no longer varies. Measured on four full-suite runs of identical code, it was not merely redundant but harmful: method per-bench med per-feature med per-feature worst raw ops/s 1.81% 2.16% 4.96% anchor-normalized 3.53% 6.23% 9.50% Dividing by the scale roughly tripled the disagreement between measurements of identical code. On one of the four runs the anchors moved -4.6% at the median while the 446 real benches moved +1.4%: the probe did not track the workload it was correcting, so its correction was injected error. No trim setting rescues a wrong central tendency. The least reproducible file in the whole suite was calibration.bench.ts itself, 9.42% against a worst real feature of 4.96%. Two facts made this safe and settled. Every committed artifact carried scale: 1 -- the apparatus was inert its whole life, so nothing published changes and there is no migration. And of ~35 comparable projects surveyed (V8, Speedometer, Node core, ts-perf, rustc-perf, LNT, SPEC, JMH, benchstat, criterion, Vue core, Deno, Bun, esbuild, tinybench, CodSpeed, ...) zero derive a host scale from a synthetic probe and divide by it. Benchmark.js shipped exactly this and removed it for exactly this reason. Deleted: calibration.bench.ts (13 frozen anchors and the never-edit-this hazard around them), calibration.test.ts, scripts/lib/calibration.ts, BASELINE_ANCHOR_HZ, TRIM, computeScale, scaleOf, and the divide-by-scale step in every consumer. Kept, because recording beats correcting: the env fingerprint moves to scripts/lib/env.ts and still travels with every artifact. If two artifacts disagree on cpu or node their numbers are not comparable, and the answer is to re-measure rather than rescale. Added instead of the correction: suiteShiftPct reports how far the whole suite moved between two artifacts, so host-wide movement is visible in one line rather than silently folded into every figure. Added scripts/verify-bench-stability.ts (pnpm metrics:verify) so the claim that this setup is trustworthy is checkable rather than asserted: it measures identical code repeatedly and fails if the results disagree by more than the apparatus allows. Thresholds are set from measured behaviour, not taste.
…the anchors The per-run governor toggle rested on a single measurement: median CV 1.79% on performance against 1.98% on powersave, with a +0.06% speed delta. That CV was computed over the calibration anchors -- the one component since shown not to track the suite it was meant to describe. The evidence for the toggle was void the moment the anchors were. What it cost: a root-owned helper on the reference host, a sudoers rule granting the bench account passwordless access to it, restore-on-signal plumbing, and a failure mode where an unrestorable governor could strand a desktop at max clocks. For an effect no longer credibly measured at all. readGovernor stays: the governor is recorded in env and warned about when it is not performance. Recording is free and useful; setting it needed root. Set it persistently in host config if wanted. The signal handler survives on its own merits, decoupled from the governor -- without it a Ctrl-C orphans a 25-minute vitest child that keeps running after its parent is gone. REVOKE ON THE REFERENCE HOST: sudo rm /etc/sudoers.d/91-v0-governor /usr/local/sbin/v0-governor
…apparatus Nine of 433 spread >20% on identical code across four runs, and they are the entire >20% tail -- the only benches that can trip a canary band without a code change. They cluster in createNested and useDate, which points at the benches rather than the machine; check them against the fixture-isolation rules before blaming the host. Also records why an rme-derived gate was rejected: correlation between log(rme) and log(cross-run spread) is 0.413, so rme explains ~17% of it. useDate's format shortDate reports rme 0.27 and still swings 14.6%. rme is within-run dispersion and does not predict between-run dispersion.
johnleider
changed the title
metrics:verify measured each sample with a single pass while its thresholds were calibrated against median-of-3 artifacts, which is what the pipeline publishes. Run on the reference host it failed on healthy hardware: per-bench median 6.00% against a 4% band, p95 26.53% against 15%. The bands were not wrong, the measurement was. Repeated passes are the largest error-reduction lever in this system -- larger than any normalization scheme -- so a single pass and a median-of-3 are not the same quantity. --merge now defaults to 3 to match, and warns when set to anything else that the bands no longer apply. Bands restated from the four median-of-3 artifacts: per-bench 3.90% median and 14.60% p95 measured, limits 6% and 20%. Verified against those four artifacts: pass, per-bench median 3.93%.
Member
Author
Verified on the reference host
433 benches across 16 features. For contrast, the same check against the four historical artifacts (which still contain the anchors): feature median 2.16%, bench median 3.93% — and the least reproducible file in that set was The verifier caught its own bug firstIts initial run failed: per-bench median 6.00% against a 4% band. That was correct behaviour and my error — it measured each sample with That failure is the point of the tool. "These numbers are trustworthy" is now a claim anyone can check in one command, rather than something asserted in a PR description. Full verification
|
The regen workflow still ran benchmarks on runs-on: ubuntu-24.04. With the calibration apparatus deleted, that combination is worse than what preceded it -- a shared runner does not pin the CPU, GHA rotates hosts across generations differing ~1.5x, and there is no longer anything compensating. The next release would have replaced reference-host numbers with whichever host the job drew. Split by whether a number cares which CPU produced it: Coverage is deterministic, so it stays on CI. The workflow now runs metrics:coverage, commits only metrics.json, and fails loudly if benchmarks.json changes -- generate-metrics.js reads that file from the working tree, so the job re-emits the committed reference-host numbers unchanged rather than measuring new ones. Benchmarks and the history series move to a documented manual runbook on the reference workstation. There is deliberately no automation: the machine is the apparatus, and no GitHub runner is that machine. This is what rustc-perf and TypeScript's ts-perf do -- dedicated hardware, not general CI. Also drops the version gate, which guarded the expensive per-version bench that no longer runs here.
The committed benchmarks.json was measured before env existed, so the log read 'Measured on unknown cpu / node ?' — which looks like a failure to read the machine rather than an artifact that never recorded one. Name the actual situation and the fix.
Member
Author
Gap found and fixed: CI was still measuring benchmarks
Split by whether a number cares which CPU produced it:
The workflow now runs Verified on the reference host: after No automation for the benchmark half is deliberate, and matches the prior art — rustc-perf and TypeScript's The regeneration runbook# On the reference workstation, from a clean checkout of master
git pull && pnpm install --frozen-lockfile
pnpm metrics # coverage + build + bench (median of 3) + metrics.json
pnpm metrics:history # only when a published version has no snapshot yet
pnpm metrics:delta --prev <previous benchmarks.json> --next apps/docs/public/benchmarks.json
pnpm metrics:checkCommit as Read the delta at feature level (identical code reproduces to ~2%); a single bench moving <10% is usually noise. If When the machine changes — new CPU, kernel, Node major, or the box is replaced — run Full docs: "Regenerating metrics" in |
johnleider
marked this pull request as ready for review
July 30, 2026 14:17
getBenchmarksData still read data.apparatus.scale and divided every quoted figure by it. With the field deleted the fallback made it a no-op, so nothing was visibly broken -- but it left dead calibration logic in the one path that feeds numbers to answers, and it would have silently resurrected if anything ever wrote an apparatus block again.
Moving benchmarks off CI removed the thing that noticed a newly published version had no snapshot. The coverage job now runs --print-missing and puts the result in the PR it already opens, so the debt is visible where someone is looking rather than dependent on remembering to check. Dry-run against the current series reports 1.0.2 as owed, which is correct.
This was referenced Jul 30, 2026
johnleider
added a commit
that referenced
this pull request
Jul 30, 2026
* chore(bench): regenerate metrics on the reference host First regeneration since #749 moved measurement off CI. Every number here was produced on the reference workstation; the series was previously measured on GitHub runners, so this is a machine change, not a code change. prev Intel Xeon 6973P-C (GHA, ci: true) next Intel i9-7980XE (reference host, idle 0.08%) The whole suite reads 39.56% slower at the median. Nothing regressed -- the Xeon is a current-generation server part and the reference box is a 2017 desktop. This is exactly the case the env fingerprint exists to make visible, and the case no scale factor can bridge. Because of that, the per-version history was re-measured with --force rather than topped up: leaving 1.0.0 and 1.0.1 on Xeon numbers while 1.0.2 came from the reference host would have put a 40% step in the middle of every sparkline. All three points now carry the same fingerprint. 1.0.2 also had no snapshot before this. Four of sixteen feature badges demote on identical code, purely because tiers are absolute thresholds and the reference machine is slower: createRegistry blazing -> fast useProxyRegistry blazing -> fast createDataGrid fast -> good createFilter fast -> good metrics.json now carries _env instead of _apparatus. * chore(bench): re-measure with the scaling governor set to performance The first pass recorded governor: powersave in every artifact, which host-guard.ts warns about — the published numbers were taken in a state the apparatus itself flags as untrustworthy. Fixed by setting the governor for the run. Two of the four tier demotions were governor artefacts and are now gone: createRegistry and useProxyRegistry are back to blazing, both having sat within a few percent of their edges. 14 of 16 badges are unchanged from the GHA-measured series. The two that remain are real. createDataGrid (Compute spans, 10,000 items: 39.5ms) and createFilter (Update query 10 times, 10,000 items: 39.3ms per op) both exceed 33.4ms — two frames at 60fps — on this host. That threshold is anchored to something a user can feel, unlike the per-item budgets, so these are the demotions worth keeping. * docs: describe the tier system that actually exists The Performance Tiers section documented a scheme replaced in #539 on 7 July: four ops/s buckets split by O(1)/O(n)/O(n2), a complexity-detection table keyed on benchmark names, and a claim that group tiers are the average of their benchmarks. None of that is true. getTier grades cost, not throughput: collection operations on per-item microseconds AND single-operation latency against frame budgets, worst axis wins; one-shot calls on call latency alone against their own budgets. getGroupTier takes the worst, never the average -- averaging is precisely what it exists to avoid, since one flattering microbench would mask a slow path. Replaces the four tier cards and the complexity table with the two budget tables the code implements, and says which thresholds are anchored to something real: 16.7ms is a frame at 60fps and 100ms is the perceptibility threshold, while the per-item and per-call budgets are conventions. Also drops 'pnpm metrics' from Running Benchmarks. It regenerates published artifacts, which now happens only on the reference machine; a reader following that instruction produces numbers CI will reject.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmarks were unreliable because GHA rotates CPUs. This fixes that by fixing the host — and deletes the calibration apparatus that was built to compensate for it, because measurement shows it made things worse.
The headline
The apparatus doubled the error it existed to remove. Four full-suite runs of identical code on the fixed host, 433 real benches, 6 run-pairs (2,598 comparisons):
Bootstrap 95% CI on the regression: +1.72pp [1.52, 1.88] — significant, not noise. Tier flips on identical code: 7 raw vs 10 normalized.
Why it could never have worked. Decomposing pairwise variance, the run-wide common shift accounts for 0.1–13%; 87–99.9% is per-bench noise no global scalar can touch. There is ~1% of host drift available to correct against a ~4% noise floor. On one of four runs the anchors read −4.6% median while the real suite ran +1.44% faster — wrong sign — and dividing by 0.9586 injected +4.32% into every published number. Trimming can't rescue it (TRIM 0/1/2/3 → 0.914 / 0.939 / 0.959 / 0.961). The least reproducible file in the whole suite was
calibration.bench.tsitself, 9.42% against a worst real feature of 4.96%.Two facts made deletion safe and obvious. Every committed artifact carried
scale: 1— the apparatus was inert its entire life, so no published number changes and there is no migration. And of ~35 comparable projects surveyed (V8, JetStream, Speedometer, Node core, TypeScriptts-perf, rustc-perf, LNT, SPEC, JMH, Gobenchstat, criterion.rs, Vue core, React, Deno, Bun, esbuild, Biome, tinybench, mitata, CodSpeed, BenchmarkDotNet, MongoDB, Mozilla), zero derive a host scale from a synthetic probe and divide by it. Benchmark.js shipped exactly this and removed it for exactly this reason. SPEC's reference machine is not precedent — it normalizes each benchmark against its own reference time, which provably cancels.The process now
ts-perf, Chrome's device lab).--runs 3, median-merged. Repeats are the only lever that measurably works: 1 run/side → 1.95% median error, 2 runs/side → 1.20%. Larger than every normalizer combined, and already in place.envfingerprint (cpu/node/busy/peak/governor) travels with every artifact. Mismatchedcpu/nodemeans not comparable → re-measure, never rescale.suiteShiftPctreports whole-suite median movement as one line, so common-mode drift is something a human reads rather than something silently folded into every figure.iowaitcounted as busy). Idle floor is 0.03–0.22% against a 5% limit, so false refusals are ~impossible and the cost of one is a re-run.pnpm metrics:verify. The claim "these numbers are trustworthy" is now checkable rather than asserted — it measures identical code repeatedly and fails if results disagree beyond measured thresholds.--runs Nto measure fresh,--from a.json b.jsonto re-check existing artifacts.Rejected, with reasons
readGovernorstays to record and warn.log(rme)vslog(cross-run spread)correlates 0.413 — explains 17%. rme is within-run dispersion and does not predict between-run dispersion.tasksetis not part of the apparatus.Honest limitations
/*#__PURE__*/that slows everything equally shows as onesuiteShiftPctline, not a gate trip. Actionable floor ~3%. No scheme tested closes this — self-normalizers make it structurally invisible, and the anchors mis-estimated it by 4.3% in the one run where it mattered.createNestedanduseDate. They are the entire >20% tail. Named in the rules doc; they look like bench bugs, not host noise.metrics:verifyis the periodic check against this.Reference-host state
The
v0-governorsudoers grant installed for the deleted toggle is now unused. It stays: narrow (two literal arguments, deliberately un-escalatable), audited, and useful for manual governor changes during operator work on the box. Nothing calls it, and nothing needs to.Verification
pnpm typecheckgreen;pnpm lintclean; 139 test files / 4657 tests pass.pnpm metrics:verifyrun against four real artifacts and against three fresh full-suite runs on the reference host.