fix(report): never render an unmeasured category as a measured zero - #62
Merged
Merged
Conversation
The integrity report presented categories it had not measured as if it had. A SessionStart hook runs in a shell and cannot introspect the model or the live tool roster; those arrive only via --live-context. With none supplied the report printed: Model : anthropic/unknown unknown Tools exposed : 0 built-in + 0 MCP server(s) tool catalog : sha256:4f53cda18c2baa0c... all three of which read as measurements. The catalog fingerprint is the hash of an empty list, identical on every run, presented in a section headed "change here == your agent changed". A reader cannot tell "we did not check" from "we checked and there is nothing", and for an integrity tool that distinction is the product. The data model already recorded this correctly: `observed` marks which categories a snapshot measured, and diff() has always scoped comparisons to it, so the diffing was never wrong. Only the rendering was. render_report now consults `observed` and labels unmeasured categories, keeps the tool-catalog line out of the fingerprint list when the roster was not measured, and qualifies a clean verdict as "nothing added, nothing subtracted in the categories checked" when coverage is partial. Servers found on disk are still shown, without being described as measured. Same fix in the Codex plugin engine, which had the same defect against its own observed categories. Also makes a bad --live-context fail loudly. It was an unhandled traceback, and the tempting alternative of treating it as absent is worse than both: the run would proceed with model, tools and MCP unmeasured while the caller believed it had supplied them, which is the false assurance this tool exists to prevent. 11 new tests. claude-code: 26 passed, up from 15. Note for reviewers: plugins/agentrust-codex/tests/test_capture.py has one pre-existing failure on main (test_signed_outputs_verify_and_pass_trace_level_zero, a TRACE conformance check on generated records). Verified failing on main with this change stashed, so it is untouched by this PR and not addressed here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The agent integrity report presented categories it had not measured as if it had. Found while investigating a real drift warning on a live install.
Why it matters
A SessionStart hook runs in a shell and cannot introspect the model or the live tool roster. Those arrive only via
--live-context. With none supplied, the report printed:The session that produced this had eight MCP servers connected. All four lines read as measurements.
4f53cda18c2baa0cis the SHA-256 of[], so the tool-catalog fingerprint is a constant, printed under a heading that says "change here == your agent changed". A reader cannot distinguish "we did not check" from "we checked and there is nothing, and for a tool whose entire proposition is nothing added, nothing subtracted, that is the product rather than a cosmetic issue.The diffing was never wrong.
observedalready records which categories a snapshot measured, anddiff()has always scoped comparisons to it, so an unmeasured category was never falsely compared. The defect was confined to rendering: the report ignoredobservedand printed defaults as facts.What changed
render_reportconsultsobservedand labels unmeasured categoriesnot measured this run, with a hint on how to include them.observedcategories.--live-contextnow fails loudly instead of raising a traceback. Treating it as absent would have been worse than either: the run would proceed with model, tools and MCP unmeasured while the caller believed it had supplied them, which is precisely the false assurance this tool exists to prevent.Test plan
claude-code/tests: 26 passed, up from 15. 11 new tests covering the unmeasured rendering path, the measured path, the qualified and unqualified verdicts, and live-context loading failures.Pre-existing failure, not from this PR.
plugins/agentrust-codex/tests/test_capture.py::test_signed_outputs_verify_and_pass_trace_level_zerofails onmain. I verified it failing with this change stashed on the same base. It is a TRACE conformance check on generated records and is untouched here.Note that
claude-code/tests/test_capture.pyandplugins/agentrust-codex/tests/test_capture.pyshare a basename with no__init__.py, so collecting both in one pytest run errors on an import-file mismatch. They pass when collected separately. Also pre-existing, and worth a follow-up if CI ever collects the repo in one pass.Not addressed
The report still refers users to
/manifest verify, which is correct for a plugin install (commands/manifest.mdships that command and is what supplies the live context). A hand-rolled install that wires the hook insettings.jsonwithout installing the commands gets advice for a command that does not exist. That is an install-completeness problem rather than an engine one, so I have left it alone here, but it is worth a line in the README about the manual path.