v0.8.43 — AgentDojo adaptive-attacker robustness (airlock as a defense)

Wires agent-airlock into AgentDojo (Debenedetti et al., NeurIPS 2024) as a defense and measures its effect on the tool_knowledge adaptive attacker — closing the repo's self-admitted "not an adaptive-attacker score" gap.

What's new

• benchmarks/agentdojo/run.py installs airlock at AgentDojo's tool-execution seam — AirlockToolsExecutor: deny-by-default least-privilege SecurityPolicy + ghost-argument BLOCK + output sanitizer. Two modes:

  • Deterministic (default; no model, no key): using AgentDojo's real tool_knowledge injection ground-truth targets on the pinned workspace + banking suites (benchmark v1.2.1), measures how many injection→task pairs airlock blocks at the tool-call seam.
  • --model <id>: the true model-in-the-loop benchmark_suite_with/without_injections for defended vs undefended (benign utility / utility-under-attack / ASR). Needs an API key.

• Result (deterministic, reported as-is): 324/384 = 84.4% of tool_knowledge injection→task pairs blocked (workspace 92.5%, banking 70.8%) — a deterministic upper bound on ASR reduction, not the model-in-the-loop ASR. The per-suite-union counterpoint (0–17%) is the honest nuance: least-privilege scoping is what does the work. Full result: benchmarks/agentdojo/RESULTS.md.

Notes

• agentdojo is a bench-only extra (pip install "agent-airlock[bench]") — never a core dependency. The airlock core stays zero-dep; the harness gates the import and skips cleanly when absent.
• No fabricated numbers. The deterministic result is computed from AgentDojo's real ground truths (verified against an independent recount). The model-in-the-loop ASR is not re-run here (no key in the build env) and is not fabricated — the --model path produces it.
• README + BENCHMARK.md gain an "Adaptive-attacker (AgentDojo)" row; the "not yet wired" admission is replaced with the real result.
• 3,312 tests pass, coverage 85.21%, green across Python 3.10–3.13.