Skip to content

v0.16.0: The judge follows --model

Choose a tag to compare

@edonadei edonadei released this 01 Oct 02:27
c2f222b

This release changes which judge grades a run by default, and saves enough on each attempt to rebuild what the judge saw.

Breaking: the judge follows --model (#225)

Without --judge-model, the judge now runs on the --model backend, using that CLI's default model, instead of always claude-code. I made this change so that people who don't use Claude Code aren't blocked: a Codex-, pi- or Hermes-only setup now needs just one CLI and no extra flag. A missing judge CLI now stops the run before any attempt is paid for.

The new default is a convenience, not a recommendation. Where you can, use a judge from a different model than the one being evaluated. A model grading its own output can be biased toward answers that look like its own. To set one, pass --judge-model <backend:model>. caliper compare now warns when two runs were graded by different judges, since part of the difference may come from a stricter or looser grader. Pass the same --judge-model to both runs to compare like for like. See ADR 0034.

Saved runs record what the judge saw (#224)

The new fields are TaskResult.expect, RunMeta.judge_prompt_version and AttemptRecord.autorater_script. render_judge_prompt is now public, so you can rebuild the exact judge prompt from a saved run. --verbose shows the expect: and the judge's script.

Built-in skill activations are shown, not scored (#223)

When a skill that ships with Claude Code fires (for example claude-api), the report now shows a "Built-in skills" line instead of reading it as nothing firing. It doesn't affect activates: scoring.

Fixes and docs

  • pi's models.json is now copied into each attempt, so custom providers work with --model pi:… (#218)
  • The backend smoke evals write inside each attempt's own folder instead of a shared /tmp file (#228). Thanks @readyagentsdev
  • The README is redesigned as a landing page with a new inbox-triage example (#227)