Skip to content

fix(report): never render an unmeasured category as a measured zero - #62

Merged
imran-siddique merged 1 commit into
mainfrom
fix/report-unmeasured-categories
Jul 31, 2026
Merged

fix(report): never render an unmeasured category as a measured zero#62
imran-siddique merged 1 commit into
mainfrom
fix/report-unmeasured-categories

Conversation

@imran-siddique

Copy link
Copy Markdown
Contributor

What

The agent integrity report presented categories it had not measured as if it had. Found while investigating a real drift warning on a live install.

Why it matters

A SessionStart hook runs in a shell and cannot introspect the model or the live tool roster. Those arrive only via --live-context. With none supplied, the report printed:

  Model          : anthropic/unknown unknown
  Tools exposed  : 0 built-in + 0 MCP server(s)
  MCP servers    : none on disk
    tool catalog      : sha256:4f53cda18c2baa0c...

The session that produced this had eight MCP servers connected. All four lines read as measurements. 4f53cda18c2baa0c is the SHA-256 of [], so the tool-catalog fingerprint is a constant, printed under a heading that says "change here == your agent changed". A reader cannot distinguish "we did not check" from "we checked and there is nothing, and for a tool whose entire proposition is nothing added, nothing subtracted, that is the product rather than a cosmetic issue.

The diffing was never wrong. observed already records which categories a snapshot measured, and diff() has always scoped comparisons to it, so an unmeasured category was never falsely compared. The defect was confined to rendering: the report ignored observed and printed defaults as facts.

What changed

  • render_report consults observed and labels unmeasured categories not measured this run, with a hint on how to include them.
  • The tool-catalog line drops out of the fingerprint list when the roster was not measured, rather than showing the hash of an empty list.
  • A clean verdict is qualified as "nothing added, nothing subtracted in the categories checked" when coverage is partial. Unqualified only when coverage is complete.
  • Servers discovered on disk are still shown, without being described as measured.
  • The same fix in the Codex plugin engine, which had the same defect against its own observed categories.
  • A bad --live-context now fails loudly instead of raising a traceback. Treating it as absent would have been worse than either: the run would proceed with model, tools and MCP unmeasured while the caller believed it had supplied them, which is precisely the false assurance this tool exists to prevent.

Test plan

  • claude-code/tests: 26 passed, up from 15. 11 new tests covering the unmeasured rendering path, the measured path, the qualified and unqualified verdicts, and live-context loading failures.
  • Both paths exercised by hand on a real install: hook-style with no live context, and enriched with this session's actual model and MCP roster. The catalog fingerprint moves off the empty-list constant once the roster is measured.

Pre-existing failure, not from this PR. plugins/agentrust-codex/tests/test_capture.py::test_signed_outputs_verify_and_pass_trace_level_zero fails on main. I verified it failing with this change stashed on the same base. It is a TRACE conformance check on generated records and is untouched here.

Note that claude-code/tests/test_capture.py and plugins/agentrust-codex/tests/test_capture.py share a basename with no __init__.py, so collecting both in one pytest run errors on an import-file mismatch. They pass when collected separately. Also pre-existing, and worth a follow-up if CI ever collects the repo in one pass.

Not addressed

The report still refers users to /manifest verify, which is correct for a plugin install (commands/manifest.md ships that command and is what supplies the live context). A hand-rolled install that wires the hook in settings.json without installing the commands gets advice for a command that does not exist. That is an install-completeness problem rather than an engine one, so I have left it alone here, but it is worth a line in the README about the manual path.

The integrity report presented categories it had not measured as if it had.
A SessionStart hook runs in a shell and cannot introspect the model or the live
tool roster; those arrive only via --live-context. With none supplied the report
printed:

  Model         : anthropic/unknown unknown
  Tools exposed : 0 built-in + 0 MCP server(s)
  tool catalog  : sha256:4f53cda18c2baa0c...

all three of which read as measurements. The catalog fingerprint is the hash of
an empty list, identical on every run, presented in a section headed "change
here == your agent changed". A reader cannot tell "we did not check" from "we
checked and there is nothing", and for an integrity tool that distinction is the
product.

The data model already recorded this correctly: `observed` marks which categories
a snapshot measured, and diff() has always scoped comparisons to it, so the
diffing was never wrong. Only the rendering was. render_report now consults
`observed` and labels unmeasured categories, keeps the tool-catalog line out of
the fingerprint list when the roster was not measured, and qualifies a clean
verdict as "nothing added, nothing subtracted in the categories checked" when
coverage is partial. Servers found on disk are still shown, without being
described as measured.

Same fix in the Codex plugin engine, which had the same defect against its own
observed categories.

Also makes a bad --live-context fail loudly. It was an unhandled traceback, and
the tempting alternative of treating it as absent is worse than both: the run
would proceed with model, tools and MCP unmeasured while the caller believed it
had supplied them, which is the false assurance this tool exists to prevent.

11 new tests. claude-code: 26 passed, up from 15.

Note for reviewers: plugins/agentrust-codex/tests/test_capture.py has one
pre-existing failure on main (test_signed_outputs_verify_and_pass_trace_level_zero,
a TRACE conformance check on generated records). Verified failing on main with
this change stashed, so it is untouched by this PR and not addressed here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant