Add code review benchmark harness

[zzwong]

Problem

cr review can now run through multiple LLM runtimes and profiles. That makes it possible to compare different model/provider setups, but there is no repeatable way to do that today.

Right now a comparison is mostly manual:

• pick a review target;
• run one profile, then another;
• save the dry-run output somewhere;
• compare findings, runtime, token usage, and cost by hand;
• remember which base/head SHA pair was tested.

That is easy to get wrong and hard to repeat after the CLI, prompts, agent profiles, or model choices change.

We should have a small benchmark harness in this repo that exercises the real cr review command in dry-run mode and writes local artifacts that can be compared across models.

Scope

Add a cr benchmark command for code review benchmarks:

• Validate a benchmark suite file before running it.
• Preflight a benchmark suite before longer runs, including selected models/cases, configured profiles, agent directories, optional cr binary path, and result directory selection.
• Run one or more benchmark cases against one or more configured cr profiles.
• Invoke the normal review flow through cr review --dry-run --json.
• Never post comments, resolve threads, request reviews, or mutate GitHub state.
• Write raw review output, stderr, normalized summary records, aggregate JSON, a run manifest, and a compact Markdown report to a local results directory.
• Support benchmark cases that pin the review URL plus base/head SHAs so old reviews can be reused as stable evaluation anchors.
• Support expected-result metadata, such as known findings or an expected clean run, so results can be compared against a baseline.
• Include grading and efficiency fields that are useful for downstream human or agent synthesis, including grade status/reason, expected-anchor counts, false-positive counts, duration, token, and cost metrics.
• Keep private benchmark cases and run outputs out of git by default.

This should stay focused on benchmarking codereview-cli behavior. It should not become a separate review daemon, a hosted evaluation service, or a checked-in leaderboard.

Acceptance Criteria

• cr benchmark validate <suite.yml> loads and validates a benchmark suite with useful errors.
• cr benchmark doctor <suite.yml> checks suite readiness without invoking model providers or running reviews.
• cr benchmark doctor supports the same selection and environment flags as runs where they are relevant: --model, --case, --results-dir, --cr-bin, and --json.
• cr benchmark run <suite.yml> runs selected cases/models through cr review --dry-run --json.
• --model, --case, --results-dir, and --cr-bin allow targeted local runs.
• Benchmark output includes enough data to compare runs: case id, model/profile, review URL, exit status, finding count, duration, token/cost metrics when available, and artifact paths.
• Run artifacts include raw review JSON, stderr, optional metrics JSON, summary.jsonl, suite-summary.json, manifest.json, and report.md.
• manifest.json records enough provenance for later comparison: tool name/version, suite path/hash, selected models/cases, run timings, and artifact paths.
• Reports and summaries include expected-baseline fields and conservative grading fields for clean cases, expected anchors, false positives, duration, token, and cost efficiency.
• Result directories and files use private permissions by default.
• Local benchmark cases and results are ignored by git, while public examples and schema/docs are safe to commit.
• README or benchmark docs explain how to define a suite, validate it, preflight it, run it, and interpret the output.
• Tests cover suite validation, command wiring, dry-run invocation arguments, preflight checks, result artifact writing, manifest generation, and report/summary rendering.
• Existing cr review behavior for normal users is unchanged.

[rianjs]

Design clarification from discussion:

For the benchmark MVP, profile should remain the configured execution/account context: Git host/auth, reviewer credentials, LLM provider/auth/adapter/credential ref, agent sources, and policy. We should not require one profile per model, because that duplicates credential/policy config and works against org rollout and secret handling.

Benchmark vocabulary should distinguish:

• Suite: a collection of benchmark cases and candidates.
• Case: one PR to run against, plus optional metadata such as expected observed base/head SHAs or anchor ranges.
• Candidate: one review configuration to try against each case, e.g. base profile + model + effort/reasoning config + agent source set + max agents/concurrency.
• Run: one candidate executed against one case.

Child issue split:

• Add safe dry-run LLM runtime overrides for benchmark candidates #96: safe dry-run LLM runtime overrides for benchmark candidates.
• Add benchmark suite schema with validate and doctor commands #97: benchmark suite schema plus validate and doctor commands.
• Implement benchmark run execution and artifact output #98: benchmark run execution and artifact output.
• Document benchmark nomenclature, metrics, and artifact conventions #99: BENCHMARKING.md, README linkage, nomenclature, metrics, and artifact conventions.

This likely needs #96 before the full harness: a safe dry-run model/effort override path for cr review, or an internal equivalent for cr benchmark. Without that, benchmarking models would require rewriting agent definitions or creating many profiles.

The MVP should measure rather than grade. Expected findings/pass-fail semantics are out of scope for now. Optional anchors should be treated as objective placement metrics only: e.g. "finding attached to this file/side/line range", not "finding was semantically correct." Reports should include anchor hit/miss counts when anchors are provided, but not pass/fail grading.

For PR SHAs, benchmark should record the observed base/head SHAs. If a suite provides expected SHAs, mismatches should be reported as warnings/fields while still running by default.

Metrics should include nullable usage fields by session/candidate/run where available: input tokens, output tokens, thinking/reasoning tokens if a provider exposes them, cache read, cache create, duration, and nullable cost only when reported by the adapter/provider. We should not bake provider pricing tables into v1.

Artifact convention:

• Shareable suites: .codereview/benchmarks/
• Private local cases/results: .cr-bench/
• Default generated results: .cr-bench/results/<suite-id>/<timestamp>/
• Path timestamps should be Windows-safe and sortable, e.g. 2026-06-03T184512Z.
• JSON manifests should store full RFC3339 timestamps, e.g. 2026-06-03T18:45:12Z.

Generated results should be ignored by default because they can contain private diffs, model output, stderr, and local paths.

[rianjs]

Completed via the child issue batch:

• Add safe dry-run LLM runtime overrides for benchmark candidates #96 added safe dry-run LLM runtime overrides for benchmark candidates.
• Add benchmark suite schema with validate and doctor commands #97 added benchmark suite schema plus validate/doctor commands.
• Implement benchmark run execution and artifact output #98 added benchmark run execution and artifact output.
• Document benchmark nomenclature, metrics, and artifact conventions #99 added BENCHMARKING.md and README coverage for terminology, schema, artifacts, privacy, metrics, and anchors.

The final implementation uses the clarified benchmark vocabulary from the design follow-up: suites, cases, candidates, and runs. Selection is by --candidate and --case; the MVP measures behavior rather than performing semantic grading.

All child issues are closed and merged, so this parent is complete.