Repository navigation
v0.18.0: Login recovery and stricter validation
I've spent some time making Caliper more reliable again. There were still quite a few ways to lose an evaluation to an authentication problem, so I worked on two things: clearer errors when a login lapses, and Caliper offering to log you back in and rerun the eval.
While I was at it, a QA pass turned up a bunch of inputs (specs, skills, MCP servers, saved runs) that slipped past validation and quietly skewed a score. Caliper now refuses those up front.
Recover from a lapsed login (#250, #253)
- When the agent's or the judge's CLI login has lapsed, the run stops with "Not logged in" (exit 2) and names the login command:
claude auth login,codex loginorhermes model. Before, a lapsed judge login graded every attempt asjudge_errorafter paying for each agent run. - In an interactive terminal, Caliper offers to run that login command and then reruns your eval. CI, agents and backgrounded runs never wait for input.
The judge fails like an attempt does (#276)
- A judge at its spending cap now stops the run (exit 2) instead of marking every remaining attempt
judge_error. - A throttled judge is retried before it gives up. Waiting between retries isn't counted as judge time.
- A malformed judge answer (valid JSON but not a verdict, such as
"passed": "false"or{}) now counts as no verdict, never as a pass or a fail (#259).
Stricter specs and skills
A spec that used to validate may now be refused. Each of these used to change a score silently:
- A key repeated in one mapping, such as two
assert:lines, where YAML kept only the last one (#260) - A blank
expect:orassert:. A whitespace-onlyassert:used to pass (#261) - A spec file name that would put runs outside its results directory (#262)
- A skill whose
SKILL.mdthe install would leave out, for example becauseforbidden_filesexcludes it (#265) - A
SKILL.mdthat is a directory or isn't UTF-8 now gets a readable error instead of a traceback (#269) - A skill's
name:is read as YAML, soname: my-skill # commentand quoted names work (#264) - MCP server and skill names must match in full.
"good\n"used to pass (#270)
More accurate measurement
- Skill snapshots record every file the install copies, so
comparereports skill drift for an edit to any file the agent could read, not justSKILL.mdand the files it links (#254) - Activation no longer counts reads of
SKILL.md.bakorSKILL.md-oldas reading the skill (#266) - The forbidden-files cheat check now catches files without an extension, such as
SECRET(#267) - MCP preflight refuses replies and tools that the agent's own client would drop, so a server can no longer pass preflight and then give the agent nothing (#271)
- MCP preflight accepts servers that speak only MCP 2026-07-28, on claude-code, which supports them (#249)
Fixes
- When
.caliper/resultsisn't a directory,runrefuses before the first attempt. If saving still fails, the run is written to a temp file instead of being lost (#272) - A results file with impossible numbers (negative tokens, a score above 1) is refused on load instead of being shown as real (#273)
- An error message that contains
[/tag]no longer crashes the error renderer (#263) - Git skill sources resolve annotated tags to their commit, and a short ref is served from the cache only when the remote confirmed it (#268)
- Hermes attempts no longer time out rebuilding a source-checkout install (#256)
- The codex judge reads its answer as UTF-8, which fixes garbled verdicts on Windows (#248)
- The callable
run()numbers unnamed tasks and refuseskbelow 1 (#274)
Skills and docs
- grill-skill names a multi-skill spec after its scenario, and reminds you to commit the spec as soon as it's written (#243, #244)