v0.11.0
The first release shaped by a real audit of this tool's own eval fixture rather than
by a code review. All three changes below were found by running 0.10.0 against
evals/golden/repo and reading what came back.
Confidence claims now ship with their base (#211)
A domain's self-assessed confidence never renders on its own. It carries how many of
that domain's rules could not be evaluated, out of how many:
self-assessed confidence high (10 of 18 rules could not be evaluated)
The smoke run produced the first self_assessment data any run had carried, and put
beside each domain's verdict distribution it showed a domain reporting high over
more than half a domain it could not check, rendering identically to one that checked
almost everything. README promises a reader the opposite.
Confidence stays self-reported and stays unchecked by the server, which is
deliberate and documented. What changed is that the claim is now always readable
against the work it was made about.
Malformed finding locations are refused on write, tolerated on read (#216)
Finding.location documents three forms: path, path:line, path:start-end. A
live run produced reports/charts.py:16,29, which is none of them and was accepted
in silence: the end-anchored suffix regex did not match, the whole string became the
path, the line numbers were never validated, and the eval scorer could not strip the
tail, so a correctly located finding scored as misplaced.
record_domain_result now refuses it and names the documented format. The check is
deliberately not on the model, because that is also what a stored run-state.json is
read through: enforcing it there would refuse every run already on disk.
Behaviour change. An auditor that records such a location now gets a hard error
where it previously passed straight through. This is why the release is minor rather
than patch.
Eval coverage 11 to 14 expectations (#215)
Three findings that the runs produced consistently, and that were not planted, are
now golden expectations: D01-R04, D05-R05, D16-R10. Each required a primary citation
that supports the claim and reproduction across both runs, and each says in its own
why that it was found and verified rather than planted. Three further candidates
were rejected on the same tests.
Verification
831 tests pass. Both recorded eval runs re-scored against the grown spec, with the
scorer controlled first each time: a run-state with every finding stripped must fail,
and does.
Full changelog: v0.10.0...v0.11.0