Skip to content

bench/: perf harness + regression-gated PERFORMANCE.md (#398)#433

Merged
InauguralPhysicist merged 1 commit into
mainfrom
feat/bench-harness-398
Jul 5, 2026
Merged

bench/: perf harness + regression-gated PERFORMANCE.md (#398)#433
InauguralPhysicist merged 1 commit into
mainfrom
feat/bench-harness-398

Conversation

@InauguralPhysicist

Copy link
Copy Markdown
Collaborator

What

Performance as an executable, regression-gated fact (Rust's credibility move) rather than a README claim.

Two metrics, on purpose

Wall-clock and instruction count answer different questions, and only one can gate CI honestly:

  • bench/run_bench.sh — wall-clock n=5 medians (the house rule). Feeds docs/PERFORMANCE.md; environment-dependent, so published, not gated.
  • bench/check_regression.sh — the gate uses cachegrind instruction counts (Ir), which are deterministic: same binary + input → identical Ir on any machine, every run. So it never flakes on runner load. In CI it compares this commit's Ir against origin/main, both built in the same runner (--vs) — a regression is a pure code-change signal, not machine noise. No cross-environment baseline drift.

Not vacuous: --selftest builds a 2×-work workload and asserts it's flagged (measured +105% Ir > 5% threshold).

The observer cost, now an executable document

observed_loop vs unobserved_loop is the same arithmetic; the difference is per-assignment observer bookkeeping — measured here at ~28% wall-clock / ~46% instructions. So unobserved: on a hot numeric loop is a real, now-gated win.

Acceptance

  • bash bench/run_bench.sh reproduces medians locally ✓
  • CI red on a deliberate 2× pessimization ✓ (the --selftest, deterministic + in-env)
  • PERFORMANCE.md regenerates from the harness ✓
  • New CI bench job; no runtime/C changes.

The MT spawn/channel cliff stays a documented contract in docs/CONCURRENCY.md; a dedicated MT workload is noted as future work.

Closes #398

🤖 Generated with Claude Code

Performance as an executable, gated fact rather than a README claim.

Two metrics, deliberately:
- bench/run_bench.sh — wall-clock n=5 medians (the house rule). These feed
  docs/PERFORMANCE.md and are environment-dependent, so they are PUBLISHED, not
  gated.
- bench/check_regression.sh — the GATE uses cachegrind instruction counts (Ir),
  which are deterministic: same binary + input => identical Ir on any machine,
  every run. So it never flakes on runner load. In CI it compares THIS commit's
  Ir against origin/main, both built in the same runner (`--vs`), so a regression
  is a pure code-change signal, not machine noise. `--selftest` builds a 2x-work
  workload and asserts it is flagged, so the gate can't rot into a vacuous pass.

Workloads: scalar loop, dict churn, string building, and observed-vs-unobserved
— which makes the observer's cost an executable document (~28% wall-clock /
~46% instructions here). New CI `bench` job. The MT spawn/channel cliff stays a
documented contract in docs/CONCURRENCY.md; a dedicated MT workload is noted as
future work.

No runtime/C changes.

Closes #398

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@InauguralPhysicist InauguralPhysicist merged commit 6c334a0 into main Jul 5, 2026
17 checks passed
@InauguralPhysicist InauguralPhysicist deleted the feat/bench-harness-398 branch July 5, 2026 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bench/: replay-pinned n=5 benchmark harness, regression-gated in CI, published as docs/PERFORMANCE.md

1 participant