Skip to content

v0.19.0: Mid-conversation tasks and CLI build tracking

Choose a tag to compare

@edonadei edonadei released this 13 Oct 20:21
0fd2631

The big change in this release: a task can now start mid-conversation. Some skills only matter several turns in, after the user has typed a slash command or the agent has already looked around. Until now, the only way to test that was to paste the earlier turns into the prompt. A task can now plant those turns as history and have the agent resume from them, on claude-code, codex and pi.

I'd been thinking for a long time about adding a simulated user, an LLM playing the person on the other side of the session. I held back because I doubted it would be useful: I worried that an LLM's idea of a user wouldn't be representative of a real one, and a score built on that would measure the wrong thing. Planted history sidesteps the problem. You can mine a trace from a real session and build the task from there. For a skill like Matt Pocock's code-review, which runs once there's already work on the table, that gives you a way to test it in real conditions: the session as it actually went, up to the point where you'd ask for the review.

Caliper also saves which CLI build ran with each result, which replaces caliper update-cli. I use Caliper heavily, and I never saw my agent use update-cli in any meaningful way. Current models already know when a CLI needs updating and how to do it. What they were missing was which build actually ran, so that's what Caliper gives them now.

Start a task mid-conversation (#292, #293, #298)

  • history: names a file of planted turns: user: turns, and assistant: turns with optional shell: calls and their output. The agent resumes from them, and prompt: is the next user turn.
  • A user turn that opens with /name of a skill in skills: is run as that slash command. Each attempt expands the installed SKILL.md the way the CLI expands a typed command, so an edit to the skill shows up in the next run. An ablated skill's command stays as typed. A /name matching no declared skill gets a warning, not a refusal, since a turn may open with a path.
  • The match is exact, case included, on every OS. /REPRO-FIRST no longer expands repro-first on macOS or Windows (#296).
  • Planted turns stay out of the attempt's transcript, so cheat detection and activates: never see them. The judge reads them as prior context and doesn't grade them.
  • Supported on claude-code, codex and pi. hermes refuses a task with history: and says so.
  • New example: examples/06-mid-conversation.

Know which CLI build ran (#299)

  • Each run saves cli_version and judge_cli_version (null when no task has expect:) in RunMeta. The version comes from the binary that actually ran, not from the first one on your PATH.
  • caliper report shows a cli row. caliper compare shows both sides and highlights a difference, so a CLI upgrade between two runs is visible.
  • When a model or an option is rejected, or a CLI is found but won't run, the error names the binary and its version: Caliper ran <path> (<version>).

Breaking changes

  • caliper update-cli is removed. Update each CLI the way you installed it (#299).
  • Caliper now runs the codex on your PATH before the one bundled with the Codex app. The bundled one can trail the CLI and reject a model your config pins. If you have both, you'll get the PATH one, and CODEX_CLI_PATH still forces either (#295).
  • The examples moved from examples/starter-pack/ to examples/. External links to the old path break (#282).

Fixes

  • A git source pinned to an abbreviated commit (ref: dbd4588f9e10) failed on a cold cache. Caliper now expands it once and caches the full commit. An ambiguous or unreachable prefix is refused with a request for the full 40-character commit (#291)
  • A git source with a relative remote (./ or ../) resolves against your directory, not the cache directory (#286)
  • A hermes config.yaml that isn't valid YAML gets a readable error instead of a traceback (#284)
  • A codex model refusal names the binary that ran, and no longer suggests --judge-model codex without a model, which reads the same pin from ~/.codex/config.toml and fails the same way (#297)

Examples and docs

  • Example 02 passed without its skill, because the prompt and notify --help both named the channel. Neither does now (#282)
  • Example 03 works with the macOS date (#282)
  • Example 05 no longer fails a receipt for containing the word "transcript" (#282)
  • Examples 02 and 05 have a real GitHub skill as a neighbour, competing for the same prompt. They need git and network on their first run (#290)
  • The git-source doc example pointed at a file that doesn't exist; it now resolves. The docs also explain when path: can be left out and that ref: should be a full commit (#290)
  • The README's "How it works" section is easier to scan (#287)

For contributors

  • codex and hermes rewrite their CLI config through one module, caliper/harness/cli_config.py (#284)
  • The results schema owns the score, pass@k and pass^k of each side of a comparison. The reporter only renders them (#285)
  • The test suite drops redundant tests, and the remaining ones observe the behaviour they claim to protect (#286)
  • The unused smoke-harness skill and the CLAUDE.md symlink are gone. Claude Code reads AGENTS.md directly (#281)