SpanGuard v6.0.0 — the bound, the gate, the loop

SpanGuard v6.0.0 — the bound, the gate, the loop

The five gaps the v5 rating named, all closed with measured artifacts.

Highlights

• Split-conformal span threshold (α=0.1): in-family over-strip bounded by construction — 0.8% on the new 500-doc extended-v6 set; the full α sweep + held-out verification published, violations out-of-family reported honestly (~15% on agent-traffic benign).
• Task-conditioned gate (292k, spanguard-taskgate): AgentDojo canonical attack escape 80.6% → 40.3% with the user's task; hand-written OOD 35.5% → 29.0%.
• Agent-loop harness: outcome columns, not guard metrics — fm-v6: NotInject utility 100% + hijack 1.3%, best of four systems (fast-v4 93.3% / 8.7%; rules 100% / 20.4%).
• Semantic escape judge: found real rule gaps the verbatim check had blessed; hardened rules replay the v3 red-team at 30/30 verbatim-stripped and 0/30 semantic escapes (both flagships).
• fm-v6 (2.23M, spanguard-fm-v6): 0.0% escape on extended-v6/extended-attacks/extended-v4, p50 20.6 ms on CPU. Retrained on CPU after the Kaggle credentials expired mid-flight — recipe and meta committed.
• int8 ONNX quantization tried and rejected on decision parity; fp32 ONNX ships.

Numbers (release run eval/results/v6-final-2021)

set	fm-v6 (tasked)	fast-v4
extended-v6 · 500 docs	0.0% escape · 0.8% over-strip	0.4% · 11.9%
hand-written OOD	29.0%	4.2%
AgentDojo-wrapped · 72 attacks	40.3%	0.0% (36.4% over-strip)
agent-loop hijack	1.3%	8.7%

Self-rating: 9.1/10 (honest rubric in docs/rating.md). Live site: https://spanguard.vercel.app

v5.0.0 — SPG-FM: one foundational model + the orchestration repair

v5.0.0 — SPG-FM: one foundational model instead of a stack

SPG-FM-1 (HF: spanguard-fm-1) — a single 1.70M char-level multi-task network trained on the full corpus-v4 union (7 languages × 18 carrier families):

• 0.0% escape on every in-domain set, in all 7 languages — agent-traffic (900 docs), extended-attacks (88 docs, all five 2026 families), extended-v4 (108 docs); doc F1 1.000 on both extended fixtures
• best payload localization in the family: span F1 0.917 / 0.901 (router stack: 0.689 / 0.612)
• hand-written OOD 10.4% escape — the v2-era ensemble number, from one 1.7M model, no router
• honest cost published: 45.4% / 26.4% over-strip on out-of-domain benign prose (doc head over-fires) — roadmap B/F attack exactly this

The orchestration repair (surfaced by the sandbox restore, fixed, re-measured):

• is_structured rebuilt to the carrier taxonomy (k=v anywhere in line, tight CSV, tabs)
• fast-v4 union-everywhere restored (every doc routes to one of three char experts)
• fm prose gating via its own document head
• fast-v4 hand-written OOD: 16.7% (v4 record) → 29.2% (broken restore) → 4.2% (repaired — all-time best for the stack); extended-attacks 7.1% → 0.0%
• broken intermediates committed (v5-fm-1504, v5-check-1514, v5-final-1519); release run v5-ship-1548

Also: measured charts (flagship bars, version ladder, 7-language matrix, cost & latency — every number traced to a committed record), world-class README, Kaggle P100 training kernel (committed, ~12k docs/min), 26 tests green.

Live: HF Space demo · models

v3.0.0 — domain expansion + honest-benchmark hardening

spanguard v3.0.0 — domain expansion + honest-benchmark hardening

New

• Corpus v3 (training/gen_corpus_v3.py): 14,000 docs, 5 languages (en/de/hu/fr/bn), 16 families — adds the 2026 attack surface: MCP tool-description poisoning (MCPTox class), agent skill/config files (Skill-Inject class), buried multi-hop payloads, markdown-image exfiltration, polite social engineering; benign traps follow red-team findings + NotInject-style FP bait. Seed-locked, hash manifest committed.
• spanguard-char-v3 (0.79M, HF: Nabidnur/spanguard-char-v3): extended-domain char guard — 0.4% escape / span F1 0.996 on the five new families vs 11.6% / 0.467 for char-v2; the old-OOD regression (handwritten 4.2% → 12.5%) is published, not hidden. Both models ship.
• spanguard-span-mdistil-v2 (135M, HF): corpus-v2 retrain with boundary-weighted loss (×3 at span edges) — public-probe over-strip 34.2% → 18.1%, hand-written over-strip 11.3% → 6.9%, escape trade 29.2% → 33.3% published as the precision/recall shift it is.
• Tool-definition scanning (gateway + spanguard tools scan|pin): OpenAI tools array / MCP descriptions scanned with rules+heuristics; rug-pull detection via operator-controlled description-hash pin file. (MCPTox: 36.5% avg poisoning success — the gateway now covers the tool-definition surface.)
• Typed markers [stripped:override] / [stripped:exfil] / … (G): the agent can tell the user what was removed. Config: SPANGUARD_TYPED_MARKERS=0 for legacy.
• Adaptive red-team loop (training/redteam_adaptive.py, idea A): LLM attacker mutates payloads against the live guard — first cut: 25 real mutations, 0 escapes in 30 attempts (eval/results/redteam-v3-0300/).

Fixed (external-review hardening)

• Evidence consistency: teacher pass stated as it ran (1,150 calls, $3.91, 0 teacher-only spans → planted ∪ teacher == planted); latency claims now quote committed p50s (fast 277 ms, char ~2 ms, rules 0.2 ms, horizon-small 87.5 ms); public-deepset reframed as a benign-only over-strip probe (escape column n/a); Wilson 95% CIs on headline numbers (char vs fast non-overlapping).
• SPEC: XLM-R 278M → distilbert 135M; test/rules counts match the code (21 tests, 18 rules); Prompt Guard 2 marked license-gated (403) with published figures cited instead of a fake head-to-head.
• Gateway: async httpx client; Bearer-by-default upstream auth (SPANGUARD_UPSTREAM_AUTH); /guard + /v1/* optional token auth; response tool-message cleaning actually walks choices[*].message; /healthz reports actually-loaded detectors + degraded reasons (no silent rules-only downgrade).
• 21 tests green, ruff 0 errors, CI badge honest.

Positioning

2026 field table (Meta PromptGuard 2 / LlamaFirewall, tool-output Sanitizer, buried-injection report, Presidio PII playbook, NotInject) + stated roadmap (task-conditioned stripping, taint-linked toolguard, conformal thresholds, NotInject/AgentDojo columns).

SpanGuard v1.0.0

SpanGuard v1.0.0 — submission release

A span-level prompt-injection guard for AI agent tool traffic: strip the attack, keep the data.

What ships

• spanguard pip package: rules engine (21 patterns) · 135M span model (BIO tagger) · audit mode · OpenAI-compatible gateway (/v1/chat/completions proxy + /guard)
• Three published models, all trained locally on CPU from one deterministic fictional corpus (en/de/hu):
  • spanguard-span-mdistil — surgical span stripping
  • spanguard-docguard-mdistil — the one-bit baseline, trained on the same data for a fair fight
  • spanguard-toolguard-mdistil — tool-call argument guard
• Committed run records + SVG charts + CI regression gate (make gate)
• Static demo Space: https://huggingface.co/spaces/Nabidnur/spanguard-demo

Headline numbers (escape % · over-strip %)

Set	spanguard fast	rules-only	protectai v3	LLM 0.5B	horizon small	horizon base
Hand-written attacks (62, en/de/hu)	29.2 · 11.3	72.9 · 7.5	35.4 · 71.8	91.7 · 13.1	8.3 · 37.7	6.2 · 59.0
Agent traffic (900, synthetic)	0.0 · 3.3	13.4 · 3.3	43.6 · 48.0	97.5 · 5.1	12.5 · 6.3	15.2 · 13.1
Public prompts (452, deepset)	0.0 · 34.2	0.0 · 0.1	0.0 · 1.2	0.0 · 5.0	0.0 · 0.0	0.0 · 0.0

Cells are escape % · over-strip %. spanguard fast is Pareto-optimal on the hand-written set;
horizon-base buys 6.2% escape by destroying 59% of benign data; a 0.5B LLM self-check misses
91.7% of hand-written attacks. Tool-call surface (400 held-out calls): toolguard blocks 100%
of attacks with 0.0% false-blocks at p50 45.5 ms; the doc-level model on the same calls
false-blocks 39.0%.

Honesty

• Fictional data only; every number is a committed, re-runnable record (eval/results/, docs/REPRODUCE.md).
• Escape and over-stripping are reported together everywhere — a guard that deletes everything escapes 0%.
• The teacher-LLM path ships as optional tooling; committed labels are deterministic planted spans. No unrun inference is claimed.
• Per-family heatmaps show which families leak for which system — including ours.

Apache-2.0. Solo project for the Meta Global AI Hackathon 2026 — Indie track.