Repository navigation
v0.16.0: The judge follows --model
This release changes which judge grades a run by default, and saves enough on each attempt to rebuild what the judge saw.
Breaking: the judge follows --model (#225)
Without --judge-model, the judge now runs on the --model backend, using that CLI's default model, instead of always claude-code. I made this change so that people who don't use Claude Code aren't blocked: a Codex-, pi- or Hermes-only setup now needs just one CLI and no extra flag. A missing judge CLI now stops the run before any attempt is paid for.
The new default is a convenience, not a recommendation. Where you can, use a judge from a different model than the one being evaluated. A model grading its own output can be biased toward answers that look like its own. To set one, pass --judge-model <backend:model>. caliper compare now warns when two runs were graded by different judges, since part of the difference may come from a stricter or looser grader. Pass the same --judge-model to both runs to compare like for like. See ADR 0034.
Saved runs record what the judge saw (#224)
The new fields are TaskResult.expect, RunMeta.judge_prompt_version and AttemptRecord.autorater_script. render_judge_prompt is now public, so you can rebuild the exact judge prompt from a saved run. --verbose shows the expect: and the judge's script.
Built-in skill activations are shown, not scored (#223)
When a skill that ships with Claude Code fires (for example claude-api), the report now shows a "Built-in skills" line instead of reading it as nothing firing. It doesn't affect activates: scoring.
Fixes and docs
- pi's
models.jsonis now copied into each attempt, so custom providers work with--model pi:…(#218) - The backend smoke evals write inside each attempt's own folder instead of a shared
/tmpfile (#228). Thanks @readyagentsdev - The README is redesigned as a landing page with a new inbox-triage example (#227)