Skip to content

bench/: replay-pinned n=5 benchmark harness, regression-gated in CI, published as docs/PERFORMANCE.md #398

Description

@InauguralPhysicist

Why. Perf claims currently live in READMEs and memory files. Rust's credibility move was making performance an executable, regression-gated fact (perf.rust-lang.org as a merge gate). We can go one better: EIGS_TRACE-pinned inputs make every benchmark run input-deterministic — a perf regression becomes a diffable artifact, not a statistical argument.

What.

  • bench/ with 5–8 workloads: scalar loop, dict-heavy, string, tensor, observed-vs-unobserved (making the ~22% unobserved win an executable document), and one spawn/channel MT case that states the Data races: parallel workers executing the same shared chunk (exec_count, JIT counters, inline caches, shared-env refcounts) #297 interpreter-fallback cliff as a measured, versioned fact. Fold in the existing tests/bench_*.eigs.
  • bench/run_bench.sh: n=5 medians (the perf-measurement house rule), workloads pinned under EIGS_TRACE.
  • CI job recording medians per commit, failing on >X% regression against a checked-in baseline.
  • docs/PERFORMANCE.md published with the same byte-checked honesty as SPEC.md.

Acceptance. bash bench/run_bench.sh reproduces medians locally; CI red on a deliberate 2x pessimization; PERFORMANCE.md numbers regenerate from the harness.

Effort: days. Source: 2026-07-04 survey (performance dimension).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions