chore(bench): record why BASELINE_ANCHOR_HZ must not be captured on GHA - #734
Closed
johnleider wants to merge 2 commits into
Closed
chore(bench): record why BASELINE_ANCHOR_HZ must not be captured on GHA#734johnleider wants to merge 2 commits into
johnleider wants to merge 2 commits into
Conversation
The doc comment on BASELINE_ANCHOR_HZ told the next maintainer to "run metrics-regen, read apparatus.anchors out of the produced benchmarks.json, paste it here". Following that literally arms the calibration unit off a GitHub Actions runner, which the 2026-07-24 dedicated-bench-host analysis had already ruled out — an in-flight GHA capture was deliberately halted rather than committed. ubuntu-24.04 pins the image, not the CPU. The pool spans ~1.9x in single-thread throughput across at least two vendors whose anchors do not differ by a scalar (0.55-0.86 per-anchor, Xeon 6973P-C to EPYC 7763), so a capture freezes an arbitrary draw, and a median of three draws just selects the majority vendor. Point both the constant and the metrics:check warning at the reference-host capture procedure in .claude/specs/2026-07-24-dedicated-bench-host.md. No behavior change: the baseline stays null, computeScale still returns 1.
…CHOR_HZ The comment told the next maintainer to paste apparatus.anchors out of a metrics-regen artifact. benchmarks.json carries a paste-ready block, so following it works and reads scale: 1.0000 — while pinning the unit to whichever GHA host that draw landed on. Point at the spec instead of restating it; the reasoning lives in .claude/specs/2026-07-24-dedicated-bench-host.md §6.
Member
Author
|
Obsolete. This documents why The concern behind this PR was right and is now enforced structurally instead of by comment: benchmarks are measured only on the fixed reference host, CI fails if it touches |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Comment-and-warning-text only.
BASELINE_ANCHOR_HZstaysnull,computeScalestill returns 1, no artifact changes.Why
The doc comment on
BASELINE_ANCHOR_HZcurrently reads:Followed literally, that arms v0's benchmark unit off a GitHub Actions runner — which
.claude/specs/2026-07-24-dedicated-bench-host.md§6 had already ruled out three days ago ("the in-flight GHA median-of-three capture was halted rather than committed"). The instruction and the decision disagree, and the instruction is the one a maintainer reads at the moment of temptation.metrics:check's warning had the same gap — it named the constant but not the host.Evidence the spec establishes, now cited at the constant:
runs-on: ubuntu-24.04pins the image, not the CPU; the pool spans ~1.9x single-thread throughput.Measured here on the committed artifacts, GHA also fails the spec's own acceptance bar (±2% aggregate, ±5% worst anchor) against itself — same job, same host, same runner image:
1.0.0.json(runs:1) vsbenchmarks.json(runs:3)1.0.1.json(runs:1) vsbenchmarks.json(runs:3)1.0.1vs1.0.0(both runs:1)Part of that is 1-run vs 3-run median noise rather than host disagreement — but
metrics:historysnapshots areruns:1, so it is the noise the baseline would actually be asked to correct.Notes
scripts/only, andchangeset-reminder.jsgates onpackages/*/src/. Ships no package version →mastertrain,choretype.node scripts/check-benchmark-artifacts.tsexits 0 and emits the amended warning; eslint clean on both files.