Skip to content

Benchmark Testing

Jump to bottom
amirbena edited this page Sep 17, 2026 · 3 revisions

Benchmark Testing

A contributor-oriented orientation page for this repository's own review-quality benchmark and its new PR-time selection/gating pipeline — not a user-facing capability of either Skill. Every rule here is owned by docs/benchmark-measurement-architecture/benchmark-measurement-architecture-model.md and docs/benchmark/ — this page links out rather than restating them, per the same canonical/Wiki boundary Feature-Pages states for feature pages.

Why it exists

Three things used to be true independently: a fixed benchmark corpus existed (docs/benchmark/corpus/), deterministic match/quality metrics existed (match-criteria.md, missed-and-incorrect-findings.md, severity-accuracy.md, duplicate-noise.md), but nothing decided which corpus cases a given PR should actually run against, or where real behavioral execution runs at all. The new pipeline (epic #329) closes that gap: a bounded, deterministic, explainable PR-time selection signal, plus two maintainer-controlled scheduled Cloud Routine lanes — a sentinel loop (the 4 permanent canonical cases, every 3 days) and a comprehensive loop (every benchmark-case/v2 fixture in the corpus tree, weekly) — each with its own independently-keyed baseline (#431), replacing the retired ad hoc PR-level classifier check.

What it does

  • Classifies each PR diff and each corpus case into one small, closed, alias-free taxonomy, then narrows candidates through a precomputed inverted index — taxonomy.md (#333).
  • Selects a bounded Top-K of the narrowed candidates by a deterministic, weighted Case Relevance Score, requiring a 60% Selection Coverage threshold and surfacing an explicit, non-silent insufficient-coverage outcome instead of a silent green check — selection.md (#334).
  • Shadow-validates the selector against broader nightly evidence (case-level miss rate, redundancy rate) before anyone trusts it — shadow-validation.md (#335).
  • Runs two scheduled lanes on maintainer-controlled Claude Cloud Routines — sentinel (the 4 fixed canonical cases, every 3 days) and comprehensive (every benchmark-case/v2 fixture in the corpus tree, derived programmatically, weekly) — each independently baselined, and manages drift per lane as a deduplicated GitHub issue lifecycle — nightly-history-and-baseline.md, drift-detection-and-regression-lifecycle.md (#338, #339, #431).
  • Retired the original, less-precise benchmark_ci_classifier.py PR-level GitHub Actions check as a redundant third path (#420) — there is no independent Actions benchmark execution path left.

When it applies

Informational, not a merge gate. This is deliberate and load-bearing, not an interim state: the pipeline runs on every applicable PR, and on two independent Cloud Routine schedules (sentinel every 3 days, comprehensive weekly, both targeting a 01:00 Israel-local start / 04:00 max-completion window that the Cloud Routine's own scheduling configuration owns — never repository runtime logic), but never blocks a normal contributor PR or merge. Real behavioral execution only ever happens through a maintainer- controlled runtime (Claude Cloud Routines) — never automatically from contributor PR automation. A future required, fail-closed gate would need a separately-decided, currently-not-pursued automatic/repository-triggered runtime class; it is not this architecture's current direction. Full rule: runtime-execution-contract.md §2.2/§4.3.

How it fits together

PR diff
  → classify into canonical taxonomy        (#333)
  → look up candidates via inverted index   (#333)
  → deterministic bounded Top-K selection   (#334)
  → require >= 60% weighted coverage        (#334)
  → (optional, maintainer-controlled) real execution against a
    qualifying Class 2 runtime — never a contributor/merge prerequisite

The scheduled branch (#338/#431 → #339) — its sentinel and comprehensive lanes running independently, each with its own baseline — runs in parallel and only rejoins this chain at #335's shadow-validation, which needs both the PR-time selector's output and the scheduled branch's classified regression history as its evidence source. The full nine-issue dependency order, the layer ownership table, and the two-execution-class runtime contract (Class 1 automatic/untrusted — currently unprovisioned and not pursued further; Class 2 maintainer-controlled — the only one in active use) are canonical in benchmark-measurement-architecture-model.md.

Where the code lives

Concern Script
Taxonomy classification + inverted index build scripts/benchmark/classify_pr_diff.py, scripts/benchmark/build_benchmark_index.py
Top-K selection scripts/benchmark/select_benchmark_cases.py
Shadow-validation scripts/benchmark/shadow_validate.py
Nightly history + drift lifecycle scripts/benchmark/benchmark_history.py, scripts/benchmark/benchmark_drift.py
Cloud Routine entrypoint scripts/benchmark/run_benchmark_routine.py

Related pages

Canonical references

Wiki Page Map

Page What it covers
Home Wiki entry point: what the two Skills are and where to go next.
Getting-Started First steps to package and run either Skill.
Choosing-a-Review-Skill How to decide between local-code-review and github-pr-review.
How-the-Review-Model-Works Concept-level walkthrough of the review pipeline shared by both Skills.
Architecture-Overview Component-level map of how the repository's pieces fit together.
Specialist-Depth-Review When and how a review goes deeper than its base pass.
Local-Code-Review-Guide Usage guide for reviewing local, not-yet-PR'd Git changes.
GitHub-PR-Review-Guide Usage guide for reviewing an existing GitHub Pull Request.
Review-Context-and-Existing-Evidence How optional requirements/context and prior review evidence are supplied.
Feature-Pages Index of per-feature capability pages and their shared template.
Feature-Delta-Re-Review How a re-review scopes itself to only what changed since the last pass.
Feature-GitHub-Publication-and-Authorization What a PR review publishes to GitHub and under what authority.
Feature-Human-Style-Review-Output The senior-engineer-voice prose summary output option.
Feature-Requirement-Coverage How supplied requirements are checked and reported as covered or not.
Feature-Runtime-Validation-Evidence How a review can execute commands or reproductions as evidence.
Severity-and-Decision-Model The P0/P1/P2 severity levels and how the merge decision is derived.
Packaging-and-Portability How the Skills go from source repository to a standalone archive.
Development-Workflow Contributor guide for working on this repository itself.
Benchmark-Testing Orientation to this repository's own review-quality benchmark pipeline.
Repository-Documentation-Map Where each concern's canonical, authoritative source lives.
AI-Code-Review-Landscape Neutral snapshot comparing this project against other AI code-review products.
FAQ Short, practical answers to common questions.

Clone this wiki locally