Skip to content

v0.18.0: Login recovery and stricter validation

Choose a tag to compare

@edonadei edonadei released this 10 Oct 06:51
290ba6c

I've spent some time making Caliper more reliable again. There were still quite a few ways to lose an evaluation to an authentication problem, so I worked on two things: clearer errors when a login lapses, and Caliper offering to log you back in and rerun the eval.

While I was at it, a QA pass turned up a bunch of inputs (specs, skills, MCP servers, saved runs) that slipped past validation and quietly skewed a score. Caliper now refuses those up front.

Recover from a lapsed login (#250, #253)

  • When the agent's or the judge's CLI login has lapsed, the run stops with "Not logged in" (exit 2) and names the login command: claude auth login, codex login or hermes model. Before, a lapsed judge login graded every attempt as judge_error after paying for each agent run.
  • In an interactive terminal, Caliper offers to run that login command and then reruns your eval. CI, agents and backgrounded runs never wait for input.

The judge fails like an attempt does (#276)

  • A judge at its spending cap now stops the run (exit 2) instead of marking every remaining attempt judge_error.
  • A throttled judge is retried before it gives up. Waiting between retries isn't counted as judge time.
  • A malformed judge answer (valid JSON but not a verdict, such as "passed": "false" or {}) now counts as no verdict, never as a pass or a fail (#259).

Stricter specs and skills

A spec that used to validate may now be refused. Each of these used to change a score silently:

  • A key repeated in one mapping, such as two assert: lines, where YAML kept only the last one (#260)
  • A blank expect: or assert:. A whitespace-only assert: used to pass (#261)
  • A spec file name that would put runs outside its results directory (#262)
  • A skill whose SKILL.md the install would leave out, for example because forbidden_files excludes it (#265)
  • A SKILL.md that is a directory or isn't UTF-8 now gets a readable error instead of a traceback (#269)
  • A skill's name: is read as YAML, so name: my-skill # comment and quoted names work (#264)
  • MCP server and skill names must match in full. "good\n" used to pass (#270)

More accurate measurement

  • Skill snapshots record every file the install copies, so compare reports skill drift for an edit to any file the agent could read, not just SKILL.md and the files it links (#254)
  • Activation no longer counts reads of SKILL.md.bak or SKILL.md-old as reading the skill (#266)
  • The forbidden-files cheat check now catches files without an extension, such as SECRET (#267)
  • MCP preflight refuses replies and tools that the agent's own client would drop, so a server can no longer pass preflight and then give the agent nothing (#271)
  • MCP preflight accepts servers that speak only MCP 2026-07-28, on claude-code, which supports them (#249)

Fixes

  • When .caliper/results isn't a directory, run refuses before the first attempt. If saving still fails, the run is written to a temp file instead of being lost (#272)
  • A results file with impossible numbers (negative tokens, a score above 1) is refused on load instead of being shown as real (#273)
  • An error message that contains [/tag] no longer crashes the error renderer (#263)
  • Git skill sources resolve annotated tags to their commit, and a short ref is served from the cache only when the remote confirmed it (#268)
  • Hermes attempts no longer time out rebuilding a source-checkout install (#256)
  • The codex judge reads its answer as UTF-8, which fixes garbled verdicts on Windows (#248)
  • The callable run() numbers unnamed tasks and refuses k below 1 (#274)

Skills and docs

  • grill-skill names a multi-skill spec after its scenario, and reminds you to commit the spec as soon as it's written (#243, #244)

For contributors

  • Each backend is declared once, in caliper/backends.py, and every surface reads from that registry (#258)
  • Engine resolution moved out of the run command into caliper/commands/engine.py (#257)