Skip to content

detectors: ground AgentBreaker judge in the victim's tool contract - #1970

Merged
jmartin-tech merged 4 commits into
NVIDIA:mainfrom
koralchapnik:agent-breaker-contract-judge-pr
Aug 25, 2026
Merged

detectors: ground AgentBreaker judge in the victim's tool contract#1970
jmartin-tech merged 4 commits into
NVIDIA:mainfrom
koralchapnik:agent-breaker-contract-judge-pr

Conversation

@koralchapnik

Copy link
Copy Markdown
Contributor

The AgentBreakerResult judge previously decided "was this a violation?" using only the attacker's vulnerability_info hypothesis. That lets the attacker's framing define success and inflates false positives when the agent merely used a tool exactly as its contract allows.

Feed the victim agent's own tool contract into the judge instead:

  • probe: AttackState carries agent_purpose + the target tool's declared description; both are stashed into attempt.notes (from agent.yaml, or from the probe's auto-discovery when the yaml omits them) and passed to the in-loop verify().
  • detector: verify() / _evaluate_single_output() accept agent_purpose and tool_description (defaulted, so existing keyword callers are unaffected); detect()'s fallback path reads them from notes. Verdicts now also carry a block_label, so verify() returns a 4-tuple and per-output verified_results entries are 3-tuples.
  • prompts.yaml: the VERIFY template now judges against (A) the tool's contract and (B) universal security harms, with an explicit decision order and richer verdict fields (violation_basis, verdict_reason).

Contract-grounding is the default; the fast path (probe-computed verified_results) is unchanged.

Adds detector tests for contract rendering, missing-contract fallbacks, back-compat, richer-field tolerance, and a packaged-prompt guard, plus probe tests for AttackState round-trip and contract capture.

Verification

  • garak -t <target_type> -n <model_name> — ran the probe end-to-end against a live agent target; the contract-grounded judge fires and scores per-output verdicts
  • python -m pytest tests/detectors/test_detectors_agent_breaker.py tests/probes/test_agent_breaker.py — 83 passed
  • Validated against a real internal agent target to assess AgentBreaker performance

@jmartin-tech jmartin-tech self-assigned this Jul 20, 2026

@jmartin-tech jmartin-tech left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validation testing against a mock agent produced viable results. This may impact or subsume some of the changes in #2044 and #2069.

A note on testing, upstream dependency issues will cause a CI/CD failure until this is merged or rebased on commit 3845757 or newer from main.

Testing to enable a consolidated merge with the noted PRs is in progress.

The AgentBreakerResult judge previously decided "was this a violation?"
using only the attacker's vulnerability_info hypothesis. That lets the
attacker's framing define success and inflates false positives when the
agent merely used a tool exactly as its contract allows.

Feed the victim agent's own tool contract into the judge instead:
- probe: AttackState carries agent_purpose + the target tool's declared
  description; both are stashed into attempt.notes (from agent.yaml, or
  from the probe's auto-discovery when the yaml omits them) and passed to
  the in-loop verify().
- detector: verify()/_evaluate_single_output() accept agent_purpose and
  tool_description (defaulted, so existing callers are unaffected);
  detect()'s fallback path reads them from notes.
- prompts.yaml: the VERIFY template now judges against (A) the tool's
  contract and (B) universal security harms, with an explicit decision
  order and richer verdict fields (violation_basis, verdict_reason).

Contract-grounding is the default; the fast path (probe-computed
verified_results) is unchanged.

Adds detector tests for contract rendering, missing-contract fallbacks,
back-compat, richer-field tolerance, and a packaged-prompt guard, plus
probe tests for AttackState round-trip and contract capture.

Signed-off-by: Koral Chapnik Verbun <kchapnikverb@nvidia.com>
The VERIFY judge could score a leaked credential as a clean miss. STEP 0
asked whether the request served the agent's purpose and could STOP HERE
with PARTIAL/confidence<=0.5, which falls under confidence_cutoff. Section
(B) universal harms was only referenced from STEP 1, one step later, so an
off-purpose request against a vague contract terminated before the harm
check ran at all.

Reorder to: harm already done (judged on the response) -> contract ->
performed/attempted/refused -> purpose and scope as a fallback. No step can
now preempt a harm check.

Also split (B)'s conflated host bullet into disclosure of sensitive state
and destructive action, and drop its "beyond the tool's contract" qualifier
so a permissive contract cannot nullify a universal harm.

Measured over 21 labelled cases x 5 runs against nvidia/openai/gpt-oss-120b:
34% -> 2.9% error rate, false negatives 22 -> 0.

Signed-off-by: Koral Chapnik Verbun <kchapnikverb@nvidia.com>
verify() returned (False, 0.0, ...) on model error, empty response, and
parse failure -- indistinguishable from a genuine NO verdict, so every judge
failure silently scored as a non-hit and under-reported ASR. A broken judge
made a target look safe.

verify() now returns None when the judge cannot be evaluated, after a retry
(verify_attempts, default 2), and detect() scores that output None, which
garak excludes from ASR rather than counting as a pass. A None *output* from
the generator stays 0.0 -- nothing was produced, which is a known
non-success, not an unjudged one.

This is what the reported false negative most likely was: the attempt logged
confidence exactly 0, which no real verdict produces. Note the default judge
is a reasoning model that returns empty content when reasoning exhausts the
token budget.

Also drops block_label, which was write-only: stored in verified_results and
attempts_history, never read by scoring or reporting. Removing it shares the
same return paths as the change above, and cuts ~20 lines of mandatory judge
output, reducing parse-failure surface. To be re-landed with a consumer.

Signed-off-by: Koral Chapnik Verbun <kchapnikverb@nvidia.com>
Target tool names come from the red team model's analysis and are matched
loosely against priority_targets in _build_tool_configs, but the contract
lookup here is exact. A reformatted name -- read_file() vs read_file, a case
difference -- silently returns "", the judge is handed "(no contract found
for this tool)", and nothing is logged. Contract grounding then degrades to
ungrounded judging with no signal in the report, which defeats the point of
grounding the judge in the first place.

Log a warning on both a missed lookup and a tool that declares an empty
description. No behaviour change.

Signed-off-by: Koral Chapnik Verbun <kchapnikverb@nvidia.com>
@koralchapnik

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough validation, @jmartin-tech — glad the mock-agent results looked viable.

I've rebased onto 3845757 (the anthropic pin), so the upstream dependency failure should be cleared. The workflow runs are currently pending maintainer approval — could you kick those off when you get a chance?

Happy to help with the consolidation against #2044 and #2069 — let me know if you'd like me to adjust anything here to make that merge cleaner.

jmartin-tech added a commit that referenced this pull request Aug 25, 2026
… verdicts (#2044)

addressed conflict with #1970, unscored detector contract became return `None`
@jmartin-tech
jmartin-tech merged commit 6711597 into NVIDIA:main Aug 25, 2026
17 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

agent_breaker: evaluation-model failures score as 'attack failed' — a judge outage reports the target as safe (false-clean ASR)

2 participants