v0.10.0

Caliper stops pasting your skill into the prompt. It installs every declared skill where the agent already looks for skills and lets the agent decide whether to load one. That one change is what the rest of this release is built on: whether a skill fires is now a thing you assert, --baseline gives way to --ablate, and a spec can name a skill straight out of a git repo.

Upgrading a spec takes about a minute. docs/MIGRATING-to-skills.md has the checklist, including the traps a find-and-replace misses.

Breaking: skill: became skills: (#80)

A spec used to name one skill under test. It now declares a set of peers, written as bare paths:

skills:
  - ./SKILL.md                     # the skill you care about
  - ../changelog-writer/SKILL.md   # a neighbour it might steal work from

The old skill: key errors with a pointer to the new shape, and caliper validate prints the replacement. A bare-agent eval drops the block entirely.

The set is closed, which is what makes activation a measurement instead of a guess. A skill you don't declare can never fire, so if yours delegates, declare the whole chain.

Breaking: nothing is preloaded (#80)

Every declared skill is installed at the backend's own skills root under its frontmatter name:, and the agent discovers it there. Gone with it: codex's invented [Skill context] prompt prepend, pi's --skill, hermes' --skills, claude-code's mangled commands/SKILL-vrd-<uid>.md, and the runner's cwd staging back-door. Backends now contribute one fact each, skills_root(ctx).

So a run now measures your description and your body at the same time. If the score drops, activates: tells you which one broke.

A lone slash-command .md is now rejected at validate (#78). No directory, no name, no description, so there is nothing for an agent to discover.

New: activates:, a third kind of check

activates: asserts the exact set of skills that loaded on an attempt.

- name: A release summary belongs to changelog-writer
  prompt: "What changed since v2.1? I need it for the release notes."
  activates: [changelog-writer]

• activates: [a] means a fired and nothing else. activates: [a, b] is how a delegating skill asserts its chain. activates: [] asserts silence.
• A task with activates: and no expect:/assert: is a trigger probe. It skips the judge entirely, so it costs a fraction of a graded task and reports as trigger only instead of a zero.
• Activation gets its own scoreboard and is never blended into the success rate. A broken description and a broken body are fixed in different places, so one number mixing them would point at neither.
• The report adds a per-skill table: how many attempts wanted each skill, how often it fired when wanted, and how often it fired when it wasn't. That last column is where you catch one skill eating another's prompts.
• New attempt outcome not_checked, for a task that authored no expect:/assert:. It leaves the denominator like an unusable attempt, but nothing went wrong, so it is never reported as an error and its tokens are not counted as wasted spend.

Detection takes the union of both observable shapes: a dedicated skill tool call, or any tool call referencing <name>/SKILL.md. Matched on the suffix, never the absolute path, because codex emits the same activation relative one run and absolute the next.

Breaking: --baseline became --ablate <name> (#81, #83)

--baseline ran a second no-skill arm inside every invocation, at full price, every time. Under install-and-discover its delta also conflated two questions: did the agent reach for the skill, and did the skill help. The activated column answers the first one already, more precisely and for free.

The observation that makes this work: an ablated arm is a property of the tasks and the surviving neighbourhood, never of the removed skill's text. That skill is not installed, so neither its body nor its description can move the number. --baseline re-measured it on every run anyway.

caliper run my-skill.eval.yaml --k 3
caliper run my-skill.eval.yaml --k 3 --ablate my-skill
caliper compare .caliper/results/my-skill/<full>.json .caliper/results/my-skill/<ablated>.json

You pay for the control arm once and re-diff it against every later iteration of the skill. --ablate is repeatable, so it generalises to leave-one-out, and naming every declared skill gets you the bare agent, which is what --baseline used to do.

• RunMeta.ablated records what was removed, so a saved run describes itself and compare labels an ablation pair from the marker instead of sniffing neighbourhood shapes.
• An ablated run drops every activation expectation rather than failing it. Remove a delegating parent and its neighbours correctly stop firing; scoring that as a miss would report the finding as a failure. The observation is still recorded and shown.
• RunResults.baseline_task_results and compare.diff_baseline are deleted. Older results JSON still loads.
• --baseline stays parseable for one release as a stub that errors with a pointer. It is not remapped, because it ran two arms and --ablate runs one.

New: git sources in skills: (#79)

skills: is now a union. A bare string is a path source and behaves exactly as it always has. A mapping is a git source that caliper clones for you:

skills:
  - ./SKILL.md
  - repo: vercel-labs/agent-skills   # or a URL, git@host:owner/name, or a local path
    ref: a1b2c3d                     # optional, omit to track the default branch
    path: skills/tdd/SKILL.md        # optional, defaults to SKILL.md at the root

Every spec in this repo declared exactly one local ./SKILL.md, so every activation number here was measured against no competition at all. Git sources let a spec name the competition without vendoring somebody's repo.

• run fetches before the first attempt, so a bad repo: costs you nothing. validate never touches the network. It resolves from a warm cache and reports the rest as not cached.
• An uncached source that cannot be fetched refuses the run, because a member silently missing would measure your skill against competition that wasn't there. A cached one whose remote is unreachable runs on the cache and says so.
• Checkouts land in ~/.cache/caliper/skills/, keyed by resolved commit, so they are immutable, shared across specs, and safe to delete. CALIPER_CACHE_DIR moves them.
• SkillSnapshot gains source_kind ("path" or "git") alongside git_repo/git_sha. Older JSON reads as "path".

An omitted ref: is allowed to move because compare now tells you when it did. It reads the per-file hashes it has been recording all along: a drifted git source warns, since the spec claimed where those bytes came from and your delta is confounded. A drifted path source is shown without alarm, because that edit is usually the reason the run exists.

 ⚠ tdd changed between runs, git source, a1b2c3d → e4f5g6h; pin `ref:` to hold it fixed
   my-skill changed between runs, path, 4fc7951 → bcbcbde

compare --format json carries skill_drift for every member, with name, source_kind, and both sides' refs, so a script sees the path-sourced ones that don't raise a warning.

Fixes

• caliper compare <spec> <spec> used to resolve both sides to the same file and diff a run against itself. Every delta read zero, has_regression was false, and no guard fired, because each guard compared the run to itself and found it consistent. --ablate makes that slip easy to reach, since the two arms are two runs of one spec sitting in one folder. The refusal is on the resolved path, so a spec name and an explicit path to the run it resolves to are caught as the same file. Two distinct runs of one spec still compare.
• validate reached the network for a commit-pinned, uncached entry, so a well-formed spec exited 1 on a plane. The cache answers first now, and the offline gate follows it.
• A hex-shaped ref was assumed to be a commit, so a branch named abcdef1 was silently pinned and never re-resolved. The remote is asked first. Only a reachable remote denying the ref proves it is a commit.
• Stale-cache warnings were collected and printed after run returned, so a run that raised lost them entirely. Those are the runs where knowing a member was stale matters most. They now go out through on_warning as they happen.
• The snapshot loop rebound its own SkillRef parameter, so the first companion file a SKILL.md pointed at turned ref into a PosixPath and every attempt died with 'PosixPath' object has no attribute 'source_kind'. Every skill using progressive disclosure crashed, which is most real ones and a shape caliper explicitly measures. The suite missed it because every snapshot fixture was a lone SKILL.md referencing nothing.
• validate now says when it could not check your activates: names instead of passing silently.

caliper list and compare

• caliper list <spec> gains an ablated column. compare needs the older side addressed by path, so anyone coming back in a later session had to tell the ablated run from the full one, and the marker was in the JSON and nowhere on screen.
• The Timestamp column is gone. Its only content was the run id in another format, and it had squeezed the id itself into an ellipsis. The File column is now Run and shows the stem, which is what report --run takes verbatim, folded rather than truncated so it survives a narrow terminal.

Internals

• caliper/attempt.py owns what an attempt means. assemble_attempt is pure over an already-produced AttemptResult, with no threads and no I/O, and the runner keeps only what an attempt costs (#85). runner.py goes from 521 lines to 369, and 20 new tests need no fake harness.
• _SkillSnapshotter moves to caliper/skillsnapshot.py, next to the fetcher that already resolved the commit. An orchestrator has no business shelling out to git rev-parse.
• New decision records: ADR 0013 (install-and-discover), 0014 (activation is a check type), 0015 (ablation names its subject at the invocation), 0016 (caliper fetches git sources itself), 0017 (unpinned git sources are allowed because drift is reported).
• Ruff pinned to 0.15.20.