Follow-up to #25 (P0.1 benchmark harness). The criterion benches/ exist and run locally (PR #40); this card adds the deterministic CI perf-regression gate that was deliberately deferred. Self-contained.
Why
Wall-clock benchmarking on shared GitHub Actions runners is structurally flaky — criterion's own author notes "differences of as much as 50% are fairly common with no change in the code at all," and CodSpeed measured a 2.66% coefficient of variation (→ ~45% false positives on a 2% gate). The fix is instruction-counting, not statistical thresholds on wall-clock numbers. See the cited analysis in the #25 research comment.
Recommendation (from #25 research)
CodSpeed — deterministic CPU-simulation (single-run, VM-noise-immune), free & unlimited for public OSI-licensed repos (this repo qualifies), and a one-line Cargo rename over the existing criterion benches (codspeed-criterion-compat) — zero bench-code changes.
Scope
Acceptance criteria
Alternatives considered (see #25)
- Self-hosted Gungraun (Valgrind instruction-count, no SaaS) — Linux-only, cannot run on the maintainer's aarch64-mac; viable if avoiding a SaaS is a hard requirement.
- Bencher — results-tracking layer (self-hostable), not a measurement engine; useful later for history/dashboards on top of Gungraun.
Pointers
Why
Wall-clock benchmarking on shared GitHub Actions runners is structurally flaky — criterion's own author notes "differences of as much as 50% are fairly common with no change in the code at all," and CodSpeed measured a 2.66% coefficient of variation (→ ~45% false positives on a 2% gate). The fix is instruction-counting, not statistical thresholds on wall-clock numbers. See the cited analysis in the #25 research comment.
Recommendation (from #25 research)
CodSpeed — deterministic CPU-simulation (single-run, VM-noise-immune), free & unlimited for public OSI-licensed repos (this repo qualifies), and a one-line Cargo rename over the existing criterion benches (
codspeed-criterion-compat) — zero bench-code changes.Scope
codspeed-criterion-compatshim (renamecriterion→ the compat crate for the bench builds, or feature-gate it).cargo codspeed build/runfor thestream-featured benches (CodSpeed runs on Linux; it is not part of the macOSnix flake check).CODSPEED_TOKENrepo secret.Acceptance criteria
nix flake checkremains green and unaffected (the perf-gate is a separate Linux job).Alternatives considered (see #25)
Pointers
benches/{matter,counter,stream}.rs; run withcargo bench --features stream.