Repository navigation
v0.19.0: Mid-conversation tasks and CLI build tracking
The big change in this release: a task can now start mid-conversation. Some skills only matter several turns in, after the user has typed a slash command or the agent has already looked around. Until now, the only way to test that was to paste the earlier turns into the prompt. A task can now plant those turns as history and have the agent resume from them, on claude-code, codex and pi.
I'd been thinking for a long time about adding a simulated user, an LLM playing the person on the other side of the session. I held back because I doubted it would be useful: I worried that an LLM's idea of a user wouldn't be representative of a real one, and a score built on that would measure the wrong thing. Planted history sidesteps the problem. You can mine a trace from a real session and build the task from there. For a skill like Matt Pocock's code-review, which runs once there's already work on the table, that gives you a way to test it in real conditions: the session as it actually went, up to the point where you'd ask for the review.
Caliper also saves which CLI build ran with each result, which replaces caliper update-cli. I use Caliper heavily, and I never saw my agent use update-cli in any meaningful way. Current models already know when a CLI needs updating and how to do it. What they were missing was which build actually ran, so that's what Caliper gives them now.
Start a task mid-conversation (#292, #293, #298)
history:names a file of planted turns:user:turns, andassistant:turns with optionalshell:calls and their output. The agent resumes from them, andprompt:is the next user turn.- A user turn that opens with
/nameof a skill inskills:is run as that slash command. Each attempt expands the installedSKILL.mdthe way the CLI expands a typed command, so an edit to the skill shows up in the next run. An ablated skill's command stays as typed. A/namematching no declared skill gets a warning, not a refusal, since a turn may open with a path. - The match is exact, case included, on every OS.
/REPRO-FIRSTno longer expandsrepro-firston macOS or Windows (#296). - Planted turns stay out of the attempt's transcript, so cheat detection and
activates:never see them. The judge reads them as prior context and doesn't grade them. - Supported on claude-code, codex and pi. hermes refuses a task with
history:and says so. - New example:
examples/06-mid-conversation.
Know which CLI build ran (#299)
- Each run saves
cli_versionandjudge_cli_version(null when no task hasexpect:) inRunMeta. The version comes from the binary that actually ran, not from the first one on yourPATH. caliper reportshows aclirow.caliper compareshows both sides and highlights a difference, so a CLI upgrade between two runs is visible.- When a model or an option is rejected, or a CLI is found but won't run, the error names the binary and its version:
Caliper ran <path> (<version>).
Breaking changes
caliper update-cliis removed. Update each CLI the way you installed it (#299).- Caliper now runs the
codexon yourPATHbefore the one bundled with the Codex app. The bundled one can trail the CLI and reject a model your config pins. If you have both, you'll get thePATHone, andCODEX_CLI_PATHstill forces either (#295). - The examples moved from
examples/starter-pack/toexamples/. External links to the old path break (#282).
Fixes
- A git source pinned to an abbreviated commit (
ref: dbd4588f9e10) failed on a cold cache. Caliper now expands it once and caches the full commit. An ambiguous or unreachable prefix is refused with a request for the full 40-character commit (#291) - A git source with a relative remote (
./or../) resolves against your directory, not the cache directory (#286) - A hermes
config.yamlthat isn't valid YAML gets a readable error instead of a traceback (#284) - A codex model refusal names the binary that ran, and no longer suggests
--judge-model codexwithout a model, which reads the same pin from~/.codex/config.tomland fails the same way (#297)
Examples and docs
- Example 02 passed without its skill, because the prompt and
notify --helpboth named the channel. Neither does now (#282) - Example 03 works with the macOS
date(#282) - Example 05 no longer fails a receipt for containing the word "transcript" (#282)
- Examples 02 and 05 have a real GitHub skill as a neighbour, competing for the same prompt. They need
gitand network on their first run (#290) - The git-source doc example pointed at a file that doesn't exist; it now resolves. The docs also explain when
path:can be left out and thatref:should be a full commit (#290) - The README's "How it works" section is easier to scan (#287)
For contributors
- codex and hermes rewrite their CLI config through one module,
caliper/harness/cli_config.py(#284) - The results schema owns the score, pass@k and pass^k of each side of a comparison. The reporter only renders them (#285)
- The test suite drops redundant tests, and the remaining ones observe the behaviour they claim to protect (#286)
- The unused smoke-harness skill and the
CLAUDE.mdsymlink are gone. Claude Code readsAGENTS.mddirectly (#281)