feat(bench): both skeptics over the same frozen passes, and the evidence bench replay dropped (#246)

[zaebee]

Tooling for #246, plus a real defect found while building it. The measurement itself did not run — see the blocker at the bottom.

The issue as written can no longer be run

#246 asks for "the bench with GUARDIAN_SKEPTIC=gemini over a mistral finder". That is production today (GUARDIAN_PROVIDER=mistral, GUARDIAN_SKEPTIC=gemini), and it is the configuration that produced 19 findings with 1 real on #399. Running it would measure what already runs and could not say why.

What was never measured is the comparison: the same finder output judged by both skeptics.

A stronger question, on the same corpus

Of the 37 frozen passes carrying findings, 30 came from a gemini finder and 7 from a mistral finder — 52 and 83 findings respectively. Both orientations are present, so judging every pass with each skeptic separates two hypotheses the issue conflates:

• cross-vendor refutes more in both orientations → the mechanism is the vendor difference, as guardian: cross-model skeptic to cut recall-lean noise without killing recall #246 supposes;
• gemini refutes more whichever finder wrote the findings → the mechanism is gemini's disposition, and "cross-model" was never the operative word.

The original single-orientation design cannot tell those apart.

Scored exactly as guardian_bench scores a run — visible_findings → match_findings → score — and carrying killed_ground_truth, because missed cannot tell "the finder never found it" from "the finder found it and the skeptic hid it" (#270), and only the second is something an arm did.

The defect this uncovered

guardian_bench's replay path hardcoded evidence=None, justified by "the replay path holds a recording, not a checkout: there is no project root". True, and beside the point — the commit is named by the fixture, so a worktree gets one, which is what guardian_replay_skeptic already does.

Production runs GUARDIAN_EVIDENCE=1. So every --replay-finder measurement to date was of a configuration nobody runs. _judge_recording now takes evidence keyword-only with no default, matching judge_all and for the same reason: this function passed None for months, and a default lets the next caller do it again silently.

Evidence covers 96 of 135 findings: four fixtures touch Python, two touch none and are judged with None because production would collect nothing for them either.

Two guards that earned their place before any call was spent

Both arms are built before either runs. Half an experiment costs the same as none and reads like a result. This is what stopped the first attempt after 2 calls instead of 135.

GUARDIAN_SKEPTIC_MODEL is dropped per arm. build_skeptic_provider applies it to whichever provider it builds, so an environment holding the production value (gemini-2.5-flash) would have handed that name to the mistral arm — one arm on a model that does not exist, the other on the intended one, and the difference reported as a comparison between vendors.

Blocker: the measurement did not run

mistral (mistral-medium-latest) -> Status 402
{"detail":"Check your subscription on https://admin.mistral.ai/subscription"}

The local MISTRAL_API_KEY has no active subscription, and no ollama is running, so no second arm exists today. gemini works.

Production is unaffected. The last review ran mistral successfully on 2026-08-16 22:53. I first read 31 occurrences of "402" in that run's log as HTTP status codes; they are the PR number, and the two "subscription" hits came from a GitHub API JSON blob. Whether the CI secret is still valid is testable only by running /guardian review.

Options, none taken here: restore the mistral subscription and run as designed; start a local ollama for a free cross-vendor arm; or use a second gemini model, which answers the weaker cross-model question rather than the cross-vendor one.

17 tests, 96% coverage on the new script; 2001 tests overall.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request updates guardian_bench.py to collect and pass evidence when replaying finder recordings instead of hardcoding it to None. It also introduces a new script, skeptic_arms.py, along with unit tests, to compare and score Gemini and Mistral skeptics on the same frozen finder passes. The review feedback highlights an issue in the _vendor classification function in skeptic_arms.py, where Mistral's codestral models are incorrectly classified as other instead of mistral, and suggests a fix to handle this case-insensitively.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
92.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud