Repository navigation
v0.42.0
agentsec v0.42.0
Offline, stdlib-only kit. 36 tools. This release adds a second independent red-team benchmark. No detector rules change.
Independent red-team round 2 (benchmarks/redteam2/)
134 new adversarial + benign look-alike samples across eleven tools (adds mcpoauth), written against v0.41.0 without reading detector source, rule files, or the round-1 fixtures (READMEs/usage and the round-1 SOURCES.md only). Techniques favor published research round 1 did not use: ShadowLeak, Policy Puppetry, Copilot autoApprove (CVE-2025-53773), CurXecute, HashJack, Gemini Trifecta, ForcedLeak, Sneaky Bits, emoji variation-selector smuggling, MPMA, MCPTox, Elastic orchestration injection, postmark-mcp, Shai-Hulud, s1ngularity AI-CLI abuse, Tracebit Gemini CLI, Replit DB deletion, EscapeRoute, mcp-remote CVE-2025-6514, MCP Inspector CVE-2025-49596, Unit 42 Bedrock memory poisoning, AgentPoison, and RFC 9700 OAuth misconfigurations. Citations: benchmarks/redteam2/SOURCES.md.
Unseen score (v0.41.0 rules)
| Tool | TP | FP | FN | TN | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|
| fetchfence | 8 | 1 | 2 | 3 | 88.9% | 80.0% | 84.2% |
| mcpoauth | 5 | 1 | 3 | 3 | 83.3% | 62.5% | 71.4% |
| mcpposture | 3 | 0 | 5 | 3 | 100.0% | 37.5% | 54.5% |
| memfence | 0 | 1 | 7 | 2 | 0.0% | 0.0% | n/a |
| pathfence | 6 | 1 | 4 | 2 | 85.7% | 60.0% | 70.6% |
| piiguard | 3 | 0 | 4 | 3 | 100.0% | 42.9% | 60.0% |
| resultguard | 2 | 2 | 9 | 2 | 50.0% | 18.2% | 26.7% |
| secretflow | 3 | 2 | 3 | 1 | 60.0% | 50.0% | 54.5% |
| shellfence | 3 | 1 | 9 | 3 | 75.0% | 25.0% | 37.5% |
| toolpoison | 0 | 0 | 8 | 4 | n/a | 0.0% | n/a |
| tracesweep | 4 | 0 | 4 | 4 | 100.0% | 50.0% | 66.7% |
| overall | 37 | 9 | 58 | 30 | 80.4% | 38.9% | 52.5% |
Round 1 for comparison: v0.40.0 unseen P=93.8% R=85.7% F1=89.6%; v0.41.0 on the same (now seen) set 100%. The round-2 drop is the honest signal that the round-1 fixes did not generalize to new techniques.
Frozen in benchmarks/baseline_redteam2_v0.41.0.json; shown on the benchmark page alongside the round-1 sections.
Detection-gap issues
#17 fetchfence, #18 mcpoauth, #19 mcpposture, #20 memfence, #21 pathfence, #22 piiguard, #23 resultguard, #24 secretflow, #25 shellfence, #26 toolpoison, #27 tracesweep
Other changes
benchmarks/run_benchmark.pyscoresmcpoauth(MO14inventory ignored) and adds the round-2 section and--checkguard.- 18 round-2 fixtures carry provider-shaped fake keys that GitHub push protection blocks; they are generated deterministically by
benchmarks/redteam2/generate.py(auto-run by the benchmark) instead of being committed. - All secrets and PII in the set are synthetic.