AgentCheck 0.5.4

A gate-correctness patch. agentcheck gate could return PASS for a target whose declared-destructive tool was never tested. It now refuses, and says which tool, which requirement, and what to do about it.

Fixed

The release gate could pass a target whose declared-destructive tool was never tested. (#91)

The gate decided from executed-case verdicts and the baseline comparison alone, and never consulted behavioral coverage. A tool you explicitly declared state_changing or destructive could therefore have zero generated cases, contribute no verdict, and leave the build green — with nothing in the output saying so.

The gate already held the principle this broke: missing evidence can never be upgraded to PASS. It applied that only to a case that ran and could not decide, never to a requirement that received no case at all. Both are the same state, and both now block.

A declaration — tool_risk in agentcheck.json, or a custom agent's own ToolDefinition — is authoritative, and makes four behaviours required for that tool: fabricated_success_after_failure, duplicate_action, ambiguous_outcome, and retry_control. If the suite produces no evidence for one, the gate blocks with exit 3, names the tool and the requirement, and says what to do next. --json carries the same under unmet_risk_obligations.

You will see: a target that declares a tool's risk but never exercises it moves from exit 0 to exit 3. That is a false green being corrected.

Scope, deliberately narrow

• Only declared risk counts. Whether an obligation exists is read from the declaration, not from what a coverage report happens to say, so risk AgentCheck merely inferred from a tool's name can never block. Inference is not authority.
• partial evidence does not block. It is the ordinary state of a healthy suite.
• An uncovered tool is not by itself a failure. success_path, failure_handling, and timeout_handling apply to every tool regardless of risk and are excluded.
• A target that declares no risk is unaffected. All four bundled examples behave exactly as before.
• FAIL, INCONCLUSIVE, and INFRA_ERROR remain distinct. An infrastructure error still outranks everything; a real behavioural failure keeps exit 1.

Two limits worth knowing

Both are named in the block message itself, not left to be discovered:

• max_cases bounds a run before obligations are evaluated, so a bounded run can report obligations unmet simply because those cases were not selected. Raise or remove it and re-run before concluding evidence is absent.
• Generation caps cases per origin. Past roughly ten declared-risky tools, AgentCheck's own generator stops emitting the cases this floor requires, and regenerating will not clear it. Do not delete a true risk declaration to go green — that trades an accurate description of your agent for a green build, which is the opposite of what the gate is for.

See the CI gate documentation.

Suite identity

GENERATOR_COMPATIBILITY_VERSION stays 1. This changes a release decision, not generation, so every suite fingerprint stays where it was.

Not changed

No new framework integration, and no expansion of what AgentCheck claims about hostile code, deterministic model execution, or complete observation.