Skip to content

v0.8.39 — block-rate comparison vs LlamaFirewall + Invariant; fix test-badge drift

Choose a tag to compare

@sattyamjjain sattyamjjain released this 29 Jun 09:33
c470e35

Added

  • bench: block-rate comparison vs LlamaFirewall + Invariant (benchmarks/blockrate/). One shared tool-call corpus (210 calls: ToolPrivBench-derived over-privileged tool selection + a public injected-tool-call set + benign controls). agent-airlock's deny-by-default presets are re-run deterministically (block-rate + p50/p95 latency); Meta LlamaFirewall and Invariant Guardrails are model-in-the-loop systems this in-process harness does not execute, so their coverage is cited, not re-run — no fabricated competitor number. Honest headline: 100% block / 0% false-positive on a self-curated corpus at p50 ~2µs/decision, contrasted categorically (not as a head-to-head score) with model-in-the-loop detection. Deterministic block-rate folds into the drift-gated BENCHMARK.md; wall-clock latency lives in benchmarks/blockrate/RESULTS.md. Run: python -m benchmarks.blockrate.

Fixed

  • test-badge version drift. The README badge sat at v0.8.5 while the package shipped v0.8.39; update_test_badge.py already read the live __version__ but had never been re-run. Badge refreshed (3,280 tests · 85.32% · v0.8.39) and a new CI guard (tests/test_badge_version.py) fails if the badge version ever diverges from the package version again. (#107)