Skip to content

chore(bench): record why BASELINE_ANCHOR_HZ must not be captured on GHA - #734

Closed
johnleider wants to merge 2 commits into
masterfrom
chore/bench-baseline-anchors
Closed

chore(bench): record why BASELINE_ANCHOR_HZ must not be captured on GHA#734
johnleider wants to merge 2 commits into
masterfrom
chore/bench-baseline-anchors

Conversation

@johnleider

Copy link
Copy Markdown
Member

What

Comment-and-warning-text only. BASELINE_ANCHOR_HZ stays null, computeScale still returns 1, no artifact changes.

Why

The doc comment on BASELINE_ANCHOR_HZ currently reads:

To capture: run metrics-regen, read apparatus.anchors out of the produced benchmarks.json, paste it here…

Followed literally, that arms v0's benchmark unit off a GitHub Actions runner — which .claude/specs/2026-07-24-dedicated-bench-host.md §6 had already ruled out three days ago ("the in-flight GHA median-of-three capture was halted rather than committed"). The instruction and the decision disagree, and the instruction is the one a maintainer reads at the moment of temptation. metrics:check's warning had the same gap — it named the constant but not the host.

Evidence the spec establishes, now cited at the constant:

  • runs-on: ubuntu-24.04 pins the image, not the CPU; the pool spans ~1.9x single-thread throughput.
  • At least two vendors, and they do not differ by a scalar: 0.55–0.86 per-anchor Xeon 6973P-C → EPYC 7763. One scale factor cannot describe that.
  • A median of three GHA runs does not help — with two vendors in three draws, the median selects the majority vendor, not a typical host.

Measured here on the committed artifacts, GHA also fails the spec's own acceptance bar (±2% aggregate, ±5% worst anchor) against itself — same job, same host, same runner image:

comparison trimmed gmean per-anchor spread worst anchor
1.0.0.json (runs:1) vs benchmarks.json (runs:3) 0.9909 0.916–1.015 a06 map build 0.916
1.0.1.json (runs:1) vs benchmarks.json (runs:3) 0.9151 0.776–1.016 a09 string build 0.776
1.0.1 vs 1.0.0 (both runs:1) 0.9274 0.805–1.006 a12 grouped aggregate 0.805

Part of that is 1-run vs 3-run median noise rather than host disagreement — but metrics:history snapshots are runs:1, so it is the noise the baseline would actually be asked to correct.

Notes

  • No changeset: scripts/ only, and changeset-reminder.js gates on packages/*/src/. Ships no package version → master train, chore type.
  • Verified: node scripts/check-benchmark-artifacts.ts exits 0 and emits the amended warning; eslint clean on both files.

The doc comment on BASELINE_ANCHOR_HZ told the next maintainer to "run
metrics-regen, read apparatus.anchors out of the produced benchmarks.json,
paste it here". Following that literally arms the calibration unit off a
GitHub Actions runner, which the 2026-07-24 dedicated-bench-host analysis
had already ruled out — an in-flight GHA capture was deliberately halted
rather than committed.

ubuntu-24.04 pins the image, not the CPU. The pool spans ~1.9x in
single-thread throughput across at least two vendors whose anchors do not
differ by a scalar (0.55-0.86 per-anchor, Xeon 6973P-C to EPYC 7763), so a
capture freezes an arbitrary draw, and a median of three draws just selects
the majority vendor.

Point both the constant and the metrics:check warning at the reference-host
capture procedure in .claude/specs/2026-07-24-dedicated-bench-host.md. No
behavior change: the baseline stays null, computeScale still returns 1.
@johnleider johnleider added this to the v1.0.x milestone Jul 28, 2026
@johnleider johnleider added the documentation Improvements or additions to documentation label Jul 28, 2026
@johnleider johnleider self-assigned this Jul 28, 2026
…CHOR_HZ

The comment told the next maintainer to paste apparatus.anchors out of a
metrics-regen artifact. benchmarks.json carries a paste-ready block, so
following it works and reads scale: 1.0000 — while pinning the unit to
whichever GHA host that draw landed on.

Point at the spec instead of restating it; the reasoning lives in
.claude/specs/2026-07-24-dedicated-bench-host.md §6.
@johnleider

Copy link
Copy Markdown
Member Author

Obsolete. This documents why BASELINE_ANCHOR_HZ must not be captured on a GHA runner; #749 deleted BASELINE_ANCHOR_HZ, computeScale, and scripts/lib/calibration.ts entirely, because measurement showed the normalization made results worse rather than better (per-bench disagreement on identical code: 1.81% raw vs 3.53% normalized).

The concern behind this PR was right and is now enforced structurally instead of by comment: benchmarks are measured only on the fixed reference host, CI fails if it touches benchmarks.json, and the reasoning is recorded under "Why there is no host calibration" in .claude/rules/benchmarks.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant