Skip to content

v0.9.1: what the tool measured, and what it was told

Choose a tag to compare

@rodlunt rodlunt released this 14 Aug 10:59
· 150 commits to main since this release
32ceeba

A follow-on to v0.9.0 from the same tester run. Both changes are about a value the report printed with more authority than it had.

Upgrading: re-run your install command with @v0.9.1. No schema change; run-states are unaffected in both directions.

The header's asserted rows now say so (#176)

Assistant and Model come from the calling assistant and cannot be checked from here, yet they rendered exactly like the rows beside them that were measured. Two runs by the same tester both executed gpt-5.6-sol while their headers read gpt-5.6-luna and GPT-5, and the two headers were initially read as evidence of a model change.

That matters more than a label: #129 defined severity as assigned by the assistant named in the header rather than measured, so the model row is what every severity in the report hangs off, and it was the one provenance row with no qualifier.

Those rows now carry a short self-reported marker, with one footnote under the grid taking the reader from the model row to the severities that depend on it. Earlier contributors inherits it, and the environment block gets the same sentence. The values themselves are unchanged: unknown-but-stated is honest, and refusing unverifiable values would only lose the information.

Started and Finished are deliberately left unmarked. The Duration row already reconciles them against the server's own clock and says so, which is a stronger statement than the qualifier.

The protocol now asks for coverage (#181)

Every domain in that run reported no coverage reported, which read as an auditor that could not be bothered. It was not. AUDIT.md had never asked: the word appeared twice in the entire protocol document, once inside the finding-body rules and once in a list of report sections, and neither was an instruction. The field existed, the report rendered a column for it, and nothing ever requested it.

It is now asked for, and it stays optional on purpose. The count cannot be verified from the server, so requiring it would guarantee a number is produced whether or not anything was counted, which is a fabricated measurement wearing the same clothes as a real one. The instruction says to omit rather than estimate, a blank keeps meaning "not claimed" rather than "claimed as zero", and the Files column carries the same qualifier as the header rows above.

Verification

797 tests passing (791 in v0.9.0). The marker is tested in both directions: present on the asserted rows, and asserted absent on Repository, Commit, Tool version, Tool commit and Duration, so it cannot quietly spread to every row and stop distinguishing anything. ruff and mypy clean.