Skip to content

v0.41.0

Choose a tag to compare

View all tags
@dunkleschitz dunkleschitz released this 10 Nov 01:07
· 14 commits to main since this release
v0.41.0
e817338

agentsec v0.41.0

Offline, stdlib-only kit. 36 tools. This release closes the independent red-team detection-gap issues. No new scanners.

Detection-gap issues #12–#16

General rule changes (not fixture-specific string matches):

  • resultguard — “disregard your prior constraints” and similar instruction nouns; conceal phrasing with words between “mention” and “the user”; private-to-public copy bait (enumerate private repos into a PR); typographic-quoted jailbreak phrases in research/discussion context stay clean; the bare word “jailbreak” in a paper or blog is no longer TR01.
  • toolpoison — “no need to mention this to the user”; mandatory shell command prefixes (Trail of Bits line-jumping); parameter names that themselves encode “read this secret file”; documentary mentions of ~/.ssh/id_rsa and similar paths stay clean when no access verb is present.
  • tracesweep — decodes base64 (and textual hex) blobs and scans the result with existing secret rules; reconstructs known-prefix secrets split across later tool-result turns.
  • shellfence — find $HOME … -exec rm / -delete and rm -rf $HOME (Amazon Q-style home wipes); DNS exfil via dig/nslookup/host to collector/OOB names or interpolated secrets; ANSI-C / hex escape normalization so $'\x72\x6d…' still hits SF02.
  • memfence — “paste your API keys into the chat” harvest wording (also closes the remaining author-corpus MF05 miss); MINJA-style query remapping and “authoritative policy for all future sessions.”
  • mcpposture — MP04 (npx -y) is suppressed when the package is already pinned to an exact version; unpinned -y still flags MP03 and MP04.

Benchmark

Set Before (v0.40.0) After (v0.41.0)
Author expanded (125 / 10 tools) 100% / 98.7% / 99.3% (75/0/1/49) 100% / 100% / 100% (76/0/0/49)
Original 85-sample, 5-tool slice 100% / 100% / 100% (51/0/0/34) unchanged
Held-out (40, written after #7–#11) 100% / 100% / 100% (26/0/0/14) unchanged
Red-team independent (100, unseen) 93.8% / 85.7% / 89.6% (60/4/10/26) frozen, still shown
Same red-team set after #12–#16 — 100% / 100% / 100% (70/0/0/30), labeled seen

The red-team set is no longer independent once these fixes landed. The original unseen v0.40.0 numbers stay on the benchmark page. A 100% seen score is not a claim about new, unpublished attacks.

python3 benchmarks/run_benchmark.py

The playground vendors for resultguard, shellfence, and mcpposture were resynced so the in-browser copies match the CLI.

The floating Action / image tag v0 points at this release (uses: dunkleschitz/agentsec@v0 or @v0.41.0).

Still not on PyPI. Publishing steps are in PUBLISHING.md.

Assets 2
Loading