Skip to content

v3.0.0 — domain expansion + honest-benchmark hardening

Choose a tag to compare

@Hisernberg Hisernberg released this 03 Nov 03:41
· 25 commits to main since this release

spanguard v3.0.0 — domain expansion + honest-benchmark hardening

New

  • Corpus v3 (training/gen_corpus_v3.py): 14,000 docs, 5 languages (en/de/hu/fr/bn), 16 families — adds the 2026 attack surface: MCP tool-description poisoning (MCPTox class), agent skill/config files (Skill-Inject class), buried multi-hop payloads, markdown-image exfiltration, polite social engineering; benign traps follow red-team findings + NotInject-style FP bait. Seed-locked, hash manifest committed.
  • spanguard-char-v3 (0.79M, HF: Nabidnur/spanguard-char-v3): extended-domain char guard — 0.4% escape / span F1 0.996 on the five new families vs 11.6% / 0.467 for char-v2; the old-OOD regression (handwritten 4.2% → 12.5%) is published, not hidden. Both models ship.
  • spanguard-span-mdistil-v2 (135M, HF): corpus-v2 retrain with boundary-weighted loss (×3 at span edges) — public-probe over-strip 34.2% → 18.1%, hand-written over-strip 11.3% → 6.9%, escape trade 29.2% → 33.3% published as the precision/recall shift it is.
  • Tool-definition scanning (gateway + spanguard tools scan|pin): OpenAI tools array / MCP descriptions scanned with rules+heuristics; rug-pull detection via operator-controlled description-hash pin file. (MCPTox: 36.5% avg poisoning success — the gateway now covers the tool-definition surface.)
  • Typed markers [stripped:override] / [stripped:exfil] / … (G): the agent can tell the user what was removed. Config: SPANGUARD_TYPED_MARKERS=0 for legacy.
  • Adaptive red-team loop (training/redteam_adaptive.py, idea A): LLM attacker mutates payloads against the live guard — first cut: 25 real mutations, 0 escapes in 30 attempts (eval/results/redteam-v3-0300/).

Fixed (external-review hardening)

  • Evidence consistency: teacher pass stated as it ran (1,150 calls, $3.91, 0 teacher-only spans → planted ∪ teacher == planted); latency claims now quote committed p50s (fast 277 ms, char ~2 ms, rules 0.2 ms, horizon-small 87.5 ms); public-deepset reframed as a benign-only over-strip probe (escape column n/a); Wilson 95% CIs on headline numbers (char vs fast non-overlapping).
  • SPEC: XLM-R 278M → distilbert 135M; test/rules counts match the code (21 tests, 18 rules); Prompt Guard 2 marked license-gated (403) with published figures cited instead of a fake head-to-head.
  • Gateway: async httpx client; Bearer-by-default upstream auth (SPANGUARD_UPSTREAM_AUTH); /guard + /v1/* optional token auth; response tool-message cleaning actually walks choices[*].message; /healthz reports actually-loaded detectors + degraded reasons (no silent rules-only downgrade).
  • 21 tests green, ruff 0 errors, CI badge honest.

Positioning

2026 field table (Meta PromptGuard 2 / LlamaFirewall, tool-output Sanitizer, buried-injection report, Presidio PII playbook, NotInject) + stated roadmap (task-conditioned stripping, taint-linked toolguard, conformal thresholds, NotInject/AgentDojo columns).