Repository navigation
SpanGuard v6.0.0 — the bound, the gate, the loop
The five gaps the v5 rating named, all closed with measured artifacts.
Highlights
- Split-conformal span threshold (α=0.1): in-family over-strip bounded by construction — 0.8% on the new 500-doc extended-v6 set; the full α sweep + held-out verification published, violations out-of-family reported honestly (~15% on agent-traffic benign).
- Task-conditioned gate (292k,
spanguard-taskgate): AgentDojo canonical attack escape 80.6% → 40.3% with the user's task; hand-written OOD 35.5% → 29.0%. - Agent-loop harness: outcome columns, not guard metrics — fm-v6: NotInject utility 100% + hijack 1.3%, best of four systems (fast-v4 93.3% / 8.7%; rules 100% / 20.4%).
- Semantic escape judge: found real rule gaps the verbatim check had blessed; hardened rules replay the v3 red-team at 30/30 verbatim-stripped and 0/30 semantic escapes (both flagships).
- fm-v6 (2.23M,
spanguard-fm-v6): 0.0% escape on extended-v6/extended-attacks/extended-v4, p50 20.6 ms on CPU. Retrained on CPU after the Kaggle credentials expired mid-flight — recipe and meta committed. - int8 ONNX quantization tried and rejected on decision parity; fp32 ONNX ships.
Numbers (release run eval/results/v6-final-2021)
| set | fm-v6 (tasked) | fast-v4 |
|---|---|---|
| extended-v6 · 500 docs | 0.0% escape · 0.8% over-strip | 0.4% · 11.9% |
| hand-written OOD | 29.0% | 4.2% |
| AgentDojo-wrapped · 72 attacks | 40.3% | 0.0% (36.4% over-strip) |
| agent-loop hijack | 1.3% | 8.7% |
Self-rating: 9.1/10 (honest rubric in docs/rating.md). Live site: https://spanguard.vercel.app