Repository navigation
v0.39.0
agentsec v0.39.0
Offline, stdlib-only kit. 35 tools, 1674 tests. This release closes the first detection-gap issues and extends the synthetic benchmark. No new scanners.
Detection-gap issues #7–#11
General rule changes (not fixture-specific string matches):
- resultguard — typo-tolerant and split-word override matching; ROT13 decode (TR06); tighter TR01 so “override the system font” stays clean; TR09 no longer flags an ordinary CDN
<img>logo, while markdown images and tracking/auto-exec URLs still fire; multilingual override phrasing. - toolpoison — soft “dump/snapshot of the workspace in the X parameter” exfil (TP04); French/Spanish/German/Portuguese/Italian overrides and “send the conversation to https://…”; common
notes/commentsparameters are only flagged when their description asks for secrets; a tool documenting its own audit BCC is no longer TP06. - tracesweep — reconstructs known secret prefixes split across trivial string concatenation (
"sk-proj-" + "BODY"). - secretflow — treats base64, hex, and URL-encoding of a previously seen secret as reuse.
- shellfence — normalizes empty quotes,
$IFS, and simple{ cmd; }wrappers so obfuscatedcurl|bashstill hits SF01.
Benchmark
On the original 85-sample, 5-tool corpus:
| Precision | Recall | F1 | TP/FP/FN/TN | |
|---|---|---|---|---|
| v0.38.0 (before) | 91.5% | 84.3% | 87.8% | 43/4/8/30 |
| v0.39.0 (after) | 100% | 100% | 100% | 51/0/0/34 |
A 40-sample held-out set was written after those rule changes were locked and was not used to tune them. Headline: precision 100%, recall 100% (26/0/0/14). That is still a small, author-written, synthetic set — not a real-world attack corpus.
The main corpus now also includes original fixtures for piiguard, fetchfence, pathfence, mcpposture, and memfence (125 samples, 10 tools): precision 100%, recall 98.7%, F1 99.3% (75/0/1/49). The remaining false negative is memfence MF05 missing a “paste your API keys into the chat” wording.
Results: dunkleschitz.github.io/agentsec/benchmarks/
python3 benchmarks/run_benchmark.pyThe playground vendors for resultguard, secretflow, and shellfence were resynced so the in-browser copies match the CLI.
The floating Action / image tag v0 points at this release (uses: dunkleschitz/agentsec@v0 or @v0.39.0).
Still not on PyPI. Publishing steps are in PUBLISHING.md.