Reliability and Benchmarks

Reliability and Benchmarks

This page explains how the repository checks that the Skills stay correct — and why that is a separate system from the review itself. Back to Architecture-Overview. For contributor-level benchmark operation, see Benchmark-Testing.

The central distinction

Deterministic tests validate contracts and small mechanisms. The benchmark is the LLM end-to-end regression layer.

A review is performed by a language model following written policies. That gives two different questions, answered by two different kinds of check:

• Are the policies and mechanisms intact? A deterministic test can answer this, the same way every time.
• Does the model, following those policies, still find the right defects? Only running a model against known cases can answer this, and the result can vary run to run.

The layers

flowchart TB
    subgraph LLM["LLM regression — scheduled, not on any request path"]
        L3["Sentinel and Comprehensive benchmark runs"] --> L4["Drift detection → sealed evidence"]
    end
    subgraph DET["Deterministic — runs on every pull request"]
        D3["Benchmark fixtures + scoring rules"]
        D2["Reference models (executable contracts)"]
        D1["Policy and governance tests"]
        D0["Unit and integration tests"]
    end
    D0 --> D1 --> D2 --> D3 --> L3

Loading

From the bottom up:

Layer	What it checks	Needs an LLM?
Unit and integration tests	The Python tooling: packaging, release, sandbox, scoring code	No
Policy and governance tests	That policy and runbook text is wired correctly — required wording, links, ordering, size limits	No
Deterministic reference models	Small test-only implementations of a contract (for example, the mechanical severity-to-decision rule), so the contract can be checked without a model	No
Benchmark fixtures and scoring	A fixed corpus of cases with expected findings, plus deterministic rules for matching a produced finding to an expected one and counting misses, false positives, severity errors, and duplicates	The scoring is deterministic; the review being scored is produced by a model
LLM benchmark	Runs the real reviewer over corpus cases and scores the output	Yes
Sentinel / Comprehensive	The two scheduled lanes: Sentinel runs a small fixed set of canonical cases frequently; Comprehensive runs the whole corpus less often	Yes
Drift and sealed evidence	Separates real regressions from run-to-run noise, confirms them, and stores each run as a sealed, content-hashed record that a publisher turns into tracking issues	Operates on the model runs' results

Where the benchmark sits

The benchmark is outside the normal review request path and outside normal pull-request CI:

• No review consults it while running. A review does not read benchmark results.
• Pull-request validation (validate.yml) runs the deterministic layers: it validates Skill metadata, builds and spec-checks the packaged Skill trees, and runs the test suite. A change-aware router picks how much of the suite runs (documentation checks only, a selected subset, everything except the integration tests, or everything, falling back to everything when unsure). It never executes the LLM benchmark.
• The LLM lanes run on a maintainer-controlled schedule in maintainer-configured Claude Cloud Routines, outside GitHub Actions. They seal each run as a content-hashed record; a separate deterministic publisher (benchmark-publish.yml, no model credentials) validates sealed records and publishes results. A deterministic, explainable selector can narrow which corpus cases a change would touch, and a shadow-validation tool can evaluate it, but both are on-demand informational tools that no CI workflow invokes.
• The benchmark is a quality-observability signal for the maintainer, not a merge prerequisite for contributors.

flowchart LR
    subgraph REQ["Review request path"]
        R1["Adapter"] --> R2["Shared engine"] --> R3["Decision"]
    end
    subgraph CI["Pull-request CI"]
        C1["Deterministic tests"]
    end
    subgraph SCHED["Scheduled, maintainer-controlled"]
        S1["LLM benchmark (Cloud Routine)"] --> S2["Drift"] --> S3["Sealed record"] --> S4["Publisher (no model credentials)"]
    end
    C1 -. "checks" .-> REQ
    S1 -. "samples" .-> REQ

Loading

Related, but not the same: telemetry

A review can record what it inspected (files, expansions, stage completion). That record is observational and never affects the Decision, and it is not benchmark ground truth. See Feature-Review-Execution-Telemetry.

Canonical references

• Tests: tests/ (see its README for the unit, policy, reference, and integration split)
• Benchmark corpus and ownership: benchmark/
• Benchmark contracts, scoring, schedule, drift, and sealing: runtime_platform/benchmark/
• Cross-component design: docs/benchmark-measurement-architecture/