Skip to content

v0.39.0

Choose a tag to compare

@dunkleschitz dunkleschitz released this 10 Nov 00:33
· 18 commits to main since this release

agentsec v0.39.0

Offline, stdlib-only kit. 35 tools, 1674 tests. This release closes the first detection-gap issues and extends the synthetic benchmark. No new scanners.

Detection-gap issues #7–#11

General rule changes (not fixture-specific string matches):

  • resultguard — typo-tolerant and split-word override matching; ROT13 decode (TR06); tighter TR01 so “override the system font” stays clean; TR09 no longer flags an ordinary CDN <img> logo, while markdown images and tracking/auto-exec URLs still fire; multilingual override phrasing.
  • toolpoison — soft “dump/snapshot of the workspace in the X parameter” exfil (TP04); French/Spanish/German/Portuguese/Italian overrides and “send the conversation to https://…”; common notes/comments parameters are only flagged when their description asks for secrets; a tool documenting its own audit BCC is no longer TP06.
  • tracesweep — reconstructs known secret prefixes split across trivial string concatenation ("sk-proj-" + "BODY").
  • secretflow — treats base64, hex, and URL-encoding of a previously seen secret as reuse.
  • shellfence — normalizes empty quotes, $IFS, and simple { cmd; } wrappers so obfuscated curl|bash still hits SF01.

Benchmark

On the original 85-sample, 5-tool corpus:

Precision Recall F1 TP/FP/FN/TN
v0.38.0 (before) 91.5% 84.3% 87.8% 43/4/8/30
v0.39.0 (after) 100% 100% 100% 51/0/0/34

A 40-sample held-out set was written after those rule changes were locked and was not used to tune them. Headline: precision 100%, recall 100% (26/0/0/14). That is still a small, author-written, synthetic set — not a real-world attack corpus.

The main corpus now also includes original fixtures for piiguard, fetchfence, pathfence, mcpposture, and memfence (125 samples, 10 tools): precision 100%, recall 98.7%, F1 99.3% (75/0/1/49). The remaining false negative is memfence MF05 missing a “paste your API keys into the chat” wording.

Results: dunkleschitz.github.io/agentsec/benchmarks/

python3 benchmarks/run_benchmark.py

The playground vendors for resultguard, secretflow, and shellfence were resynced so the in-browser copies match the CLI.

The floating Action / image tag v0 points at this release (uses: dunkleschitz/agentsec@v0 or @v0.39.0).

Still not on PyPI. Publishing steps are in PUBLISHING.md.