Skip to content

P0.1b · Wire CodSpeed CI perf-gate #41

Description

@joeldsouzax

Follow-up to #25 (P0.1 benchmark harness). The criterion benches/ exist and run locally (PR #40); this card adds the deterministic CI perf-regression gate that was deliberately deferred. Self-contained.

Why

Wall-clock benchmarking on shared GitHub Actions runners is structurally flaky — criterion's own author notes "differences of as much as 50% are fairly common with no change in the code at all," and CodSpeed measured a 2.66% coefficient of variation (→ ~45% false positives on a 2% gate). The fix is instruction-counting, not statistical thresholds on wall-clock numbers. See the cited analysis in the #25 research comment.

Recommendation (from #25 research)

CodSpeed — deterministic CPU-simulation (single-run, VM-noise-immune), free & unlimited for public OSI-licensed repos (this repo qualifies), and a one-line Cargo rename over the existing criterion benches (codspeed-criterion-compat) — zero bench-code changes.

Scope

  • Add the codspeed-criterion-compat shim (rename criterion → the compat crate for the bench builds, or feature-gate it).
  • Add a separate Ubuntu GitHub Actions workflow running cargo codspeed build/run for the stream-featured benches (CodSpeed runs on Linux; it is not part of the macOS nix flake check).
  • Create a CodSpeed account for the repo + add the CODSPEED_TOKEN repo secret.
  • Configure the regression threshold and make the PR status check blocking.

Acceptance criteria

  • A CodSpeed status check runs on PRs and reports per-benchmark deltas.
  • A deliberate slowdown in a hot path is caught by the gate (verify once).
  • nix flake check remains green and unaffected (the perf-gate is a separate Linux job).

Alternatives considered (see #25)

  • Self-hosted Gungraun (Valgrind instruction-count, no SaaS) — Linux-only, cannot run on the maintainer's aarch64-mac; viable if avoiding a SaaS is a hard requirement.
  • Bencher — results-tracking layer (self-hostable), not a measurement engine; useful later for history/dashboards on top of Gungraun.

Pointers

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformancePerformance / zero-copy worktestingTest harnesses, fuzzing, differential testing

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions