Skip to content

v0.8.35 — ToolPrivBench-style least-privilege block-rate benchmark

Choose a tag to compare

View all tags
@sattyamjjain sattyamjjain released this 22 Jun 08:25
v0.8.35
d339fa2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Added

add ToolPrivBench-style least-privilege block-rate benchmark (public, MIT, re-runnable)

benchmarks/toolprivbench/ measures how agent-airlock's deny-by-default + least-privilege SecurityPolicy handles over-privileged tool selection, mapped to the OWASP Agentic Top-10. Anchor: ToolPrivBench / "When Lower Privileges Suffice" (arXiv:2606.20023) — 8 domains, 5 risk patterns, with the transient-failure amplifier.

  • Per scenario: SecurityPolicy(default_deny=True, allowed_tools=[low_priv_tool]) → records over-priv BLOCKED (check_tool_allowed raises PolicyViolation), blocked after transient failure (the amplifier where ToolPrivBench shows prompt-level controls degrade), and low-priv ALLOWED (precision, not a blunt deny-all). No model, no network — deterministic.
  • Consumes the official dataset when present ($TOOLPRIVBENCH_DATA / data/toolprivbench.json); else a labelled subset harness (~20/pattern, 100 total).
  • RESULTS.md (regen: python -m benchmarks.toolprivbench --write): block-rate per risk pattern + per OWASP id, transient-failure column, best-effort crosswalk, and the honest caveat — measures runtime BLOCK behaviour under fixed presets, not model behaviour. Last run 2026-06-22: 100% over-priv blocked (incl. under transient failure), 100% low-priv allowed.
  • 8 smoke tests run the harness end-to-end. MIT — no paid tier, license gate, or hosted dashboard. (#103)

Source: https://arxiv.org/abs/2606.20023

Assets 2
Loading