v0.12.0
Nine changes, most of them found by the tool being pointed at its own eval
harness rather than by review.
The eval spec can now say "either answer is acceptable" (#213, #199)
expect gains a third state, finding-or-not-applicable, with its own outcome,
ruled-out. It exists for a rule where the fixture genuinely admits two
defensible readings, so raising the finding and ruling the rule out both count.
It still refuses pass, could-not-evaluate and no verdict at all: those are the
auditor not engaging. The line is that a not-applicable verdict has had to carry
a note naming the precondition that does not hold since schema version 4, so it is
a position someone wrote down. could-not-evaluate is the shrug.
ruled-out exits clean but is counted and printed separately from hit, so a
score resting partly on the permissive state cannot read identically to one that
found everything outright. Exactly one expectation uses it.
EVAL_RESULT_SCHEMA_VERSION is 3.
Updating the pin now has a command that works (#219)
The old instruction was "change the tag and re-register", and it failed:
claude mcp add refuses to overwrite an existing name. Anyone following it hit an
error, and the natural recovery leaves them silently on the old pinned build,
which is the whole failure the pinning exists to prevent.
The path genuinely differs per host, and all three are now documented separately,
with the CLI behaviours verified live rather than assumed:
| host | update |
|---|---|
| Claude Code | remove, then add; refuses to overwrite |
| Codex | re-add; it replaces in place |
| Gemini | edit the tag in settings.json; no CLI step exists |
--scope user is documented too. Without it claude mcp add defaults to local
scope and registers the server for one directory only.
The stale-update check now says what to do about it
It detected a stale build and reported it, then stopped, while the remedy lived in
documentation that was itself wrong. It now names the command that clears it, keyed
on the host that asked. An unrecognised host gets a documentation pointer, never a
guessed command: a wrong command is worse than none, because a user who runs it and
sees no error concludes they have updated when they have not.
The summary-reads-clean guard covers the whole class (#189)
That guard was written after the fourth instance of a rendered summary reading
clean over a gap, and scoped itself to <summary>. The defect is not scoped to
<summary>, and both known blind spots have since been hit: a count in an <h3>,
and a table cell, which is where the fifth instance lived. It now tracks the
"N of M" idiom wherever it renders, across two modules.
Every report image rebuilt (#225)
The screenshots had drifted two generations behind the report they are pictures
of, still showing a Confidence column without the base clause the last release
added. The social card embeds one of them and was stale twice over.
A currency test now fails the suite when any of them falls behind again. It needed
fetch-depth: 0 in CI: on the default shallow clone every path resolves to the
same commit, so the comparison held trivially and the guard reported success while
checking nothing. That is the exact defect class the guard was written to catch,
occurring inside the guard, and only a strict xfail surfaced it.
The capture method now lives in docs/social-card/README.md rather than in a
commit message, which is why this went stale twice rather than once.
Verification
847 tests pass. Both recorded eval runs re-scored, with the scorer controlled
first each time: a run-state with every finding stripped must fail, and does.
Full changelog: v0.11.0...v0.12.0