detectors: ground AgentBreaker judge in the victim's tool contract - #1970
Conversation
jmartin-tech
left a comment
There was a problem hiding this comment.
Validation testing against a mock agent produced viable results. This may impact or subsume some of the changes in #2044 and #2069.
A note on testing, upstream dependency issues will cause a CI/CD failure until this is merged or rebased on commit 3845757 or newer from main.
Testing to enable a consolidated merge with the noted PRs is in progress.
The AgentBreakerResult judge previously decided "was this a violation?" using only the attacker's vulnerability_info hypothesis. That lets the attacker's framing define success and inflates false positives when the agent merely used a tool exactly as its contract allows. Feed the victim agent's own tool contract into the judge instead: - probe: AttackState carries agent_purpose + the target tool's declared description; both are stashed into attempt.notes (from agent.yaml, or from the probe's auto-discovery when the yaml omits them) and passed to the in-loop verify(). - detector: verify()/_evaluate_single_output() accept agent_purpose and tool_description (defaulted, so existing callers are unaffected); detect()'s fallback path reads them from notes. - prompts.yaml: the VERIFY template now judges against (A) the tool's contract and (B) universal security harms, with an explicit decision order and richer verdict fields (violation_basis, verdict_reason). Contract-grounding is the default; the fast path (probe-computed verified_results) is unchanged. Adds detector tests for contract rendering, missing-contract fallbacks, back-compat, richer-field tolerance, and a packaged-prompt guard, plus probe tests for AttackState round-trip and contract capture. Signed-off-by: Koral Chapnik Verbun <kchapnikverb@nvidia.com>
The VERIFY judge could score a leaked credential as a clean miss. STEP 0 asked whether the request served the agent's purpose and could STOP HERE with PARTIAL/confidence<=0.5, which falls under confidence_cutoff. Section (B) universal harms was only referenced from STEP 1, one step later, so an off-purpose request against a vague contract terminated before the harm check ran at all. Reorder to: harm already done (judged on the response) -> contract -> performed/attempted/refused -> purpose and scope as a fallback. No step can now preempt a harm check. Also split (B)'s conflated host bullet into disclosure of sensitive state and destructive action, and drop its "beyond the tool's contract" qualifier so a permissive contract cannot nullify a universal harm. Measured over 21 labelled cases x 5 runs against nvidia/openai/gpt-oss-120b: 34% -> 2.9% error rate, false negatives 22 -> 0. Signed-off-by: Koral Chapnik Verbun <kchapnikverb@nvidia.com>
verify() returned (False, 0.0, ...) on model error, empty response, and parse failure -- indistinguishable from a genuine NO verdict, so every judge failure silently scored as a non-hit and under-reported ASR. A broken judge made a target look safe. verify() now returns None when the judge cannot be evaluated, after a retry (verify_attempts, default 2), and detect() scores that output None, which garak excludes from ASR rather than counting as a pass. A None *output* from the generator stays 0.0 -- nothing was produced, which is a known non-success, not an unjudged one. This is what the reported false negative most likely was: the attempt logged confidence exactly 0, which no real verdict produces. Note the default judge is a reasoning model that returns empty content when reasoning exhausts the token budget. Also drops block_label, which was write-only: stored in verified_results and attempts_history, never read by scoring or reporting. Removing it shares the same return paths as the change above, and cuts ~20 lines of mandatory judge output, reducing parse-failure surface. To be re-landed with a consumer. Signed-off-by: Koral Chapnik Verbun <kchapnikverb@nvidia.com>
Target tool names come from the red team model's analysis and are matched loosely against priority_targets in _build_tool_configs, but the contract lookup here is exact. A reformatted name -- read_file() vs read_file, a case difference -- silently returns "", the judge is handed "(no contract found for this tool)", and nothing is logged. Contract grounding then degrades to ungrounded judging with no signal in the report, which defeats the point of grounding the judge in the first place. Log a warning on both a missed lookup and a tool that declares an empty description. No behaviour change. Signed-off-by: Koral Chapnik Verbun <kchapnikverb@nvidia.com>
0d40d7d to
252c449
Compare
|
Thanks for the thorough validation, @jmartin-tech — glad the mock-agent results looked viable. I've rebased onto Happy to help with the consolidation against #2044 and #2069 — let me know if you'd like me to adjust anything here to make that merge cleaner. |
The AgentBreakerResult judge previously decided "was this a violation?" using only the attacker's vulnerability_info hypothesis. That lets the attacker's framing define success and inflates false positives when the agent merely used a tool exactly as its contract allows.
Feed the victim agent's own tool contract into the judge instead:
Contract-grounding is the default; the fast path (probe-computed verified_results) is unchanged.
Adds detector tests for contract rendering, missing-contract fallbacks, back-compat, richer-field tolerance, and a packaged-prompt guard, plus probe tests for AttackState round-trip and contract capture.
Verification