v0.48.0

agentsec v0.48.0

Offline, stdlib-only kit. 38 tools. This release generalizes detectors by class for independent red-team round 8 (issues #28–#36), then re-scores that set as seen. The original v0.47.0 unseen score stays the headline independent number. 2085 unit tests.

Generalization (class-level, not fixture-specific instances)

Round 8 is the development set. Rules were extended around classes, checked on a private labeled dev set of new variants, then frozen before the round-8 fixtures were re-scored:

• resultguard / toolpoison — a concealed-document channel covering charset and markup that hides text: UTF-7, HTML/Markdown/LaTeX comment and conditional blocks, hidden or tiny CSS, Mermaid/PlantUML remote-image labels, and structured clinical/office narratives (FHIR/CDA/OOXML vanished text), with decode-and-rescan. Documentation that merely mentions UTF-7 or ordinary clinical text stays clean.
• toolpoison — annotation-vs-behavior contradictions (readOnlyHint / destructiveHint / a read-only title vs. destructive verbs in the name, description, or schema), including tokenized names such as drop_namespace. Defensive mentions of sensitive paths by a tool whose purpose is to manage them, and third-person discussion of attack phrases, are no longer TP03/TP01 false positives.
• shellfence — reverse tunnels (ssh/autossh -R, RemoteForward) and container/namespace escapes (nsenter -t 1, chroot /proc/1/root, unshare plus proc root) as a class. Ordinary ssh login and -L local forwards stay clean.
• memfence — memories that change trust or identity, pre-approve money movement, weaken security controls (2FA/MFA/confirmation), or route private data to an external sink (public issue/repo, SFTP/SCP export), including instructions buried in a style preference. “2FA is enabled” and “do not auto-approve payments without confirmation” stay clean.
• fetchfence — cloud-provider metadata hostnames as a class (metadata.<provider-domain>, IMDS paths), not a single Equinix hostname.
• pathfence — local messaging/mail databases (Messages, Mail, WhatsApp ChatStorage, and similar) and infra-access / backup-key stores (Teleport tsh, Borg, restic, kopia, and similar) as path classes.
• mcpposture — unpinned installers in more ecosystems (gem exec, mise/rtx/asdf exec, WSL-wrapped launchers), container runtime sockets (containerd.sock, crio.sock, podman.sock as well as docker.sock), and proxy-environment hijack (HTTP_PROXY / HTTPS_PROXY / ALL_PROXY to a non-loopback host).
• secretflow — push-notification sinks as a class (ntfy, Pushover, Gotify, OneSignal, and similar).
• mcpoauth — look-alike authorization servers (typo/confusable of the resource host or of a well-known issuer) and over-broad first-party scopes (*.all, mail.send, cloud-platform). Narrow scopes such as Mail.Read stay clean. A well-known identity provider listed for a different app host is not treated as a look-alike.

Benchmark

The headline independent number is still round 8 on frozen v0.47.0 detectors. Unseen F1 across successive independent rounds: 78.3 → 74.4 → 61.5 → 81.3 → 96.1 → 78.2.

Set	P / R / F1	Notes
Author corpus (125)	100% / 100% / 100%	76/0/0/49
Held-out	100% / 100% / 100%	26/0/0/14
Red-team round 1 (v0.40.0, unseen)	93.8% / 85.7% / 89.6%	60/4/10/26, frozen
Same round-1 set after later fixes	100% / 100% / 100%	70/0/0/30, labeled seen
Red-team round 2 (v0.41.0, unseen)	80.4% / 38.9% / 52.5%	37/9/58/30, frozen
Same round-2 set after later generalization	100% / 100% / 100%	95/0/0/39, labeled seen
Red-team round 3 (v0.43.0, unseen)	97.3% / 65.5% / 78.3%	36/1/19/28, frozen
Same round-3 set after later generalization	100% / 100% / 100%	55/0/0/29, labeled seen
Red-team round 4 (v0.44.0, unseen)	97.0% / 60.4% / 74.4%	32/1/21/26, frozen
Same round-4 set after later generalization	100% / 100% / 100%	53/0/0/27, labeled seen
Red-team round 5 (v0.45.0, unseen)	96.0% / 45.3% / 61.5%	24/1/29/26, frozen
Same round-5 set after later generalization	100% / 100% / 100%	53/0/0/27, labeled seen
Red-team round 6 (v0.46.0, unseen)	97.4% / 69.8% / 81.3%	37/1/16/26, frozen
Same round-6 set after later generalization	100% / 100% / 100%	53/0/0/27, labeled seen
Red-team round 7 (v0.47.0, unseen)	100.0% / 92.5% / 96.1%	49/0/4/27, scored once
Red-team round 8 (v0.47.0, unseen)	95.6% / 66.2% / 78.2%	43/2/22/33, frozen; headline independent
Same round-8 set after v0.48.0 generalization	100.0% / 93.8% / 96.8%	61/0/4/35, labeled seen

Round 8 (benchmarks/redteam8/, 100 samples) was written against the frozen v0.47.0 detectors by a separate author who did not read tool source, tests, or prior fixtures. Detection-gap issues #28–#36 describe the misses. Fixes were designed from the issue text and a private dev set; the fixtures were not opened while designing rules, then the set was re-scored once as seen. Remaining honest seen misses: a FHIR narrative and a LaTeX iffalse channel that the concealed-document class did not extract, a trust/identity memory whose wording the new class did not match, and a push-notification hop whose tool name the sink class did not treat as outbound. Those four were left as-is. Closed #28–#36.

Issues

Closed #28, #29, #30, #31, #32, #33, #34, #35, #36. Good-first-issues #1–#5 were not touched.

v0.47.0

agentsec v0.47.0

Offline, stdlib-only kit. 38 tools. This release continues the class-level approach that reversed the unseen-F1 decline in v0.46.0, then scores a fresh round-7 set once. 2043 unit tests.

Generalization (class-level, not fixture-specific instances)

Round 6 is the development set. Rules were extended around classes, then checked on a private labeled dev set of new variants, then frozen before any round-7 sample was written:

• toolpoison / resultguard — an encoding-concealment layer covering MIME/RFC 2047 encoded-words and base-N transforms (Base45/58/85 and similar) with decode-and-rescan, plus Unicode confusable / mixed-script detection using a UTS #39-style skeleton (NFKD + lookalike fold). Documentation that merely mentions an encoding, and ordinary foreign-script place names, stay clean.
• shellfence — interpreter one-liners (-c/-e/-r) and generic network-relay / upload tools (socat, ncat, wget --post-file, curl --upload-file / --data-binary @file) as a class, only when a network or home/secret sink is present. A bare python -c 'print(sys.version)', wget of public docs, or a local LISTEN+FILE server is not in this class.
• memfence — memories that schedule a future data transfer, add a recipient or channel (BCC/CC, status text, webhook), or silently rewrite user input / destinations / queries. Negation must modify the action (so "do not recognize" and "ask before cc'ing" stay clean).
• pathfence — config-path patterns for CLI tools that store tokens in a hidden user dir (~/.<tool>/ or ~/.config/<app>/ plus token- or config-ish basenames), excluding known dev-tool apps.
• mcpposture — unpinned remote code execution as a class across ecosystems (nix flake refs, cargo install, go, npx, uvx, pipx, deno, docker/podman :latest). Wildcard binds (--bind *, --listen *, bare *) are treated as bind-all.
• fetchfence — documentation/preview paths on loopback (/docs, /playground, and similar) are not SSRF; loopback metadata, Consul, Docker API, gopher, and /admin still flag.
• tracesweep — incoming webhook URLs as a secret class (Slack, Discord, Teams, and generic /api/webhooks/<token>), not only xox* bot tokens.

Benchmark

Unseen F1 on successive independent rounds had been 78.3 → 74.4 → 61.5, then recovered to 81.3 on round 6 after the first class-level rewrite. This release is scored against a new blind round 7. The series is now:

Set	P / R / F1	Notes
Author corpus (125)	100% / 100% / 100%	76/0/0/49
Held-out	100% / 100% / 100%	26/0/0/14
Red-team round 1 (v0.40.0, unseen)	93.8% / 85.7% / 89.6%	60/4/10/26, frozen
Same round-1 set after later fixes	100% / 100% / 100%	70/0/0/30, labeled seen
Red-team round 2 (v0.41.0, unseen)	80.4% / 38.9% / 52.5%	37/9/58/30, frozen
Same round-2 set after later generalization	100% / 100% / 100%	95/0/0/39, labeled seen
Red-team round 3 (v0.43.0, unseen)	97.3% / 65.5% / 78.3%	36/1/19/28, frozen
Same round-3 set after later generalization	100% / 100% / 100%	55/0/0/29, labeled seen
Red-team round 4 (v0.44.0, unseen)	97.0% / 60.4% / 74.4%	32/1/21/26, frozen
Same round-4 set after later generalization	100% / 100% / 100%	53/0/0/27, labeled seen
Red-team round 5 (v0.45.0, unseen)	96.0% / 45.3% / 61.5%	24/1/29/26, frozen
Same round-5 set after later generalization	100% / 100% / 100%	53/0/0/27, labeled seen
Red-team round 6 (v0.46.0, unseen)	97.4% / 69.8% / 81.3%	37/1/16/26, frozen
Same round-6 set after v0.47.0 generalization	100% / 100% / 100%	53/0/0/27, labeled seen
Red-team round 7 (v0.47.0, unseen)	100.0% / 92.5% / 96.1%	49/0/4/27, scored once

Round 7 (benchmarks/redteam7/, 80 samples) was written after the rules were frozen, from published techniques rounds 1–6 did not use (Jupyter markdown cells, GraphQL SDL descriptions, SRT captions, Open Graph og:description, systemd Description=, svn log, Apple plist strings, RDF Turtle comments, JSON Schema default, Base58, quoted-printable, MCP _meta, fullwidth confusables, lua -e / curl --data-binary / ncat / osascript / tar|curl / openssl s_client, forget-archive and clipboard-rewrite memory, Spanish DNI / Irish PPSN / Chinese resident ID / Croatian OIB / Belgian NISS, ldap:// / Docker Engine API / ftp:// / [::1]/server-status / /cgi-bin/, glab/heroku/sfdx/linode/netlify paths, pipx run / podman :latest / --listen * / nix profile install / deno jsr:, Notion / PagerDuty / Loki sinks, shpat_ / github_pat_ / vercel_ tokens, and HTTP introspection / userinfo / PAR endpoints; see SOURCES.md). Numbers are reported honestly. No detector was changed after this set existed.

Remaining round-7 misses (honest, not retuned): shellfence tar | curl --data-binary and openssl s_client relays, a memfence logout/PagerDuty-note transfer, and a pathfence Linode CLI config.ini. The v0.46.0 loopback-docs fetchfence false positive is gone, and the round-5 seen leftovers (memfence query-rewrite and emergency-CC, tracesweep Slack incoming webhook) now flag.

The floating Action / image tag v0 points at this release (uses: dunkleschitz/agentsec@v0 or @v0.47.0).

v0.46.0

agentsec v0.46.0

Offline, stdlib-only kit. 38 tools (foundry #38 promptaudit shipped on main earlier the same day). This release changes approach: class-level detectors instead of enumerating missed instances, then scores a fresh round-6 set once. 2010 tests.

Generalization (class-level, not fixture-specific instances)

Round 5 is the development set. Rules were rewritten around classes, then checked on a private labeled dev set of new variants, then frozen before any round-6 sample was written:

• toolpoison / resultguard — a generic obfuscation layer: text that is not natural language (encoded or transformed text, low dictionary-word ratio, unusual scripts, leftover control characters, overstrike) is treated as suspicious, plus a best-effort decode-and-rescan for common transforms (including hex dumps). All MCP catalog surfaces are walked generically (prompts, prompt arguments, prompt messages, resources, templates, and server instructions). Well-formed ANSI color codes are typography, not concealment.
• piiguard — a data-driven national-ID table (30+ country formats) with checksum validators and context keywords. Unlabeled ambiguous digit strings require keywords so order numbers and build metrics stay clean; distinctive formats (letters, separators, reserved prefixes) may still fire on checksum alone.
• pathfence — a generic sensitive-path classifier: credential/key/token/vault/state/keyring/wallet/session/profile-db name patterns, known credential-bearing app dirs, XDG/Application Support roots, and cluster config basenames — not a list of the missed paths.
• shellfence — a generic rule for execution or exfil via exec-hook options (-exec, xargs, $(shell, interpreter system() or a transfer tool targeting home or secrets, plus a broader LOLBin/GTFOBins-style table.
• mcpoauth / mcpposture — the whole OAuth/OIDC metadata document is checked against spec-level requirements (token_endpoint_auth_methods_supported=none, wildcard or non-https redirects, http endpoints on any *_uri field). Generic launch hygiene: floating @latest / :latest / ==latest in any ecosystem, cwd of /, binding to all interfaces.
• secretflow — local comparison tools (equals, assert_eq, and compare-shaped left/right arguments) are not treated as reuse or exfil.

Benchmark

Unseen F1 on successive independent rounds had been falling (78.3 → 74.4 → 61.5). This release is scored against a new blind round 6. The series is now:

Set	P / R / F1	Notes
Author corpus (125)	100% / 100% / 100%	76/0/0/49
Held-out	100% / 100% / 100%	26/0/0/14
Red-team round 1 (v0.40.0, unseen)	93.8% / 85.7% / 89.6%	60/4/10/26, frozen
Same round-1 set after later fixes	100% / 100% / 100%	70/0/0/30, labeled seen
Red-team round 2 (v0.41.0, unseen)	80.4% / 38.9% / 52.5%	37/9/58/30, frozen
Same round-2 set after later generalization	100% / 100% / 100%	95/0/0/39, labeled seen
Red-team round 3 (v0.43.0, unseen)	97.3% / 65.5% / 78.3%	36/1/19/28, frozen
Same round-3 set after later generalization	100% / 100% / 100%	55/0/0/29, labeled seen
Red-team round 4 (v0.44.0, unseen)	97.0% / 60.4% / 74.4%	32/1/21/26, frozen
Same round-4 set after later generalization	100% / 100% / 100%	53/0/0/27, labeled seen
Red-team round 5 (v0.45.0, unseen)	96.0% / 45.3% / 61.5%	24/1/29/26, frozen
Same round-5 set after v0.46.0 generalization	100% / 94.3% / 97.1%	50/0/3/27, labeled seen
Red-team round 6 (v0.46.0, unseen)	97.4% / 69.8% / 81.3%	37/1/16/26, scored once

Round 6 (benchmarks/redteam6/, 80 samples) was written after the rules were frozen, from published techniques rounds 1–5 did not use (iCalendar DESCRIPTION, vCard NOTE, WebVTT, OpenAPI info.description, YAML front matter, RFC 2047 encoded-words, JSON-LD, Base45, JSON Schema examples/title, mixed-script confusables, python -c / socat / bitsadmin / node -e / php -r / wget --post-file, farewell-dump and planted-proxy memory, typo-correction and SAR/BCC memory, Slack status, Finnish hetu / Swiss AHV / Taiwan NID / HKID / Argentine CUIL, smb:// / IPv4-mapped IPv6 / tftp / Consul / AWS IMDS IPv6, rclone.conf / containers auth.json / flyctl / Windows Credential Manager / Pulumi credentials, nix run / pnpm dlx / cargo install / bind * / cwd C:\, Discord / SMTP / Jira sinks, glpat- / sk_live_ / npm_ tokens, implicit grant, and HTTP revocation_endpoint / registration_endpoint; see SOURCES.md). Numbers are reported honestly. No detector was changed after this set existed.

Remaining round-6 misses: RFC 2047 and Base45 concealment, Cyrillic/Latin confusables, python -c / socat / wget --post-file transfer/exec shapes, four new memory classes (farewell dump, typo-correction rewrite, SAR/BCC, Slack status), rclone.conf and flyctl paths, and unpinned nix run / cargo install plus --bind *. One false positive: a documented loopback docs preview flagged by fetchfence. Remaining round-5 seen misses (not in this release's scope): memfence query-rewrite and emergency-CC, and a tracesweep Slack incoming webhook.

The floating Action / image tag v0 points at this release (uses: dunkleschitz/agentsec@v0 or @v0.46.0).

agentsec v0.45.0

agentsec v0.45.0

Offline, stdlib-only kit. 37 tools. This release generalizes detection for the misses in red-team round 4, then scores a fresh round-5 set once. No new scanners. 1893 tests.

Generalization (not fixture-specific string matches)

Round 4 is the development set. Rules were extended around attack class before any round-5 sample was written:

• resultguard — strips ANSI CSI / conceal views (same fg/bg, SGR 8, erase) and flags hidden instructions; broader exfil phrasing (workspace listing, transcript, messages); first-person / prefilled-assistant-turn bait ("I will now call", "next assistant message should"). TR10 policy-frame authority now needs a second signal, so a vendor rate-limit XML stays clean while Policy Puppetry interaction-config still flags.
• toolpoison — scans MCP resourceTemplates / resources and instructions-only docs; decode layers now include ROT13 and reversed-text views.
• shellfence — SF19 LOLBin trampoline / cloud-sync exfil (tar --checkpoint-action=exec, forfiles /c, MSBuild PostBuildEvent, rclone of $HOME without --dry-run); wrapped commands are re-scanned for embedded payloads.
• memfence — MF13 auto-approve / confirmation-bypass for privileged tools (honors negation, so "never auto-approve… always confirm first" stays clean); MF14 impossible user-age claims (≤2 or ≥100). Structurally impossible identity claims only; worldview claims are out of scope.
• piiguard — PG05 also covers Polish PESEL, Singapore NRIC/FIN, French NIR, Japanese My Number, and Mexican CURP, with checksums where the format defines them (NIR / My Number / CURP are labeled-format).
• pathfence — PF15 credential-store paths (pass, 1Password, GNOME keyring, KeePass, and similar); PF16 privilege-escalation write content (ALL=(ALL) NOPASSWD, authorized_keys, ld.so.preload).
• mcpposture — flags Deno / Bun / Node / Python running an unpinned remote http(s) URL as a script (MP11+MP23). Local deno run ./local.ts stays clean.
• tracesweep — AGE-SECRET-KEY-1 Bech32-ish secret keys (TS-AGE-SECRET, critical).
• mcpoauth — MO15 critical when signing-alg lists contain none (JWT alg=none).
• secretflow — official-provider host allowlist by longest key prefix (exact host or subdomain only; lookalikes such as api.openai.com.evil.test still fire).

Benchmark

Lead with the latest unseen score, not the seen ones.

Set	Score (P / R / F1)	Counts (tp/fp/fn/tn)
Author expanded (125 / 10 tools)	100% / 100% / 100%	76/0/0/49
Original 85-sample, 5-tool slice	100% / 100% / 100%	51/0/0/34
Held-out (40, written after #7–#11)	100% / 100% / 100%	26/0/0/14
Red-team round 1 independent (v0.40.0, unseen)	93.8% / 85.7% / 89.6%	60/4/10/26, frozen, still shown
Same round-1 set after later generalization	100% / 100% / 100%	70/0/0/30, labeled seen
Red-team round 2 independent (v0.41.0, unseen)	80.4% / 38.9% / 52.5%	37/9/58/30, frozen, still shown
Same round-2 set after v0.45.0 generalization	100% / 100% / 100%	95/0/0/39, labeled seen
Red-team round 3 independent (v0.43.0, unseen)	97.3% / 65.5% / 78.3%	36/1/19/28, frozen, still shown
Same round-3 set after later generalization	100% / 100% / 100%	55/0/0/29, labeled seen
Red-team round 4 independent (v0.44.0, unseen)	97.0% / 60.4% / 74.4%	32/1/21/26, frozen, still shown
Same round-4 set after v0.45.0 generalization	100% / 100% / 100%	53/0/0/27, labeled seen
Red-team round 5 (v0.45.0, unseen)	96.0% / 45.3% / 61.5%	24/1/29/26, scored once

Round 5 (benchmarks/redteam5/, 80 samples) was written after the rules were frozen, from published techniques rounds 1–4 did not use (CSV HYPERLINK, DNS TXT, git log, JSON \uXXXX escapes, Atom/RSS, decoded JWT claims, ASCII backspace overstrike, SAML AttributeValue, MCP prompts, Base32/Braille/Atbash, find -exec, schtasks, rsync $HOME, make $(shell), wmic, awk system(), planted API base-URL / query-rewrite / language-pref / emergency-CC / locale-override memory, TCKN / RUT / South African ID / codice fiscale / BSN, gopher / hex / octal / LDAP / Kubernetes-DNS SSRF, vault-token / age keys.txt / FileZilla / terraform.tfstate / Signal config paths, go run @latest, npx github:, cwd /, uvx git+http, bind 0.0.0.0, Telegram / S3 / GraphQL sinks, pypi / Slack webhook / Groq keys, token_endpoint_auth_method=none, wildcard redirects, cleartext frontchannel_logout_uri, and others; see SOURCES.md). Numbers are reported honestly. Remaining misses are new encodings and catalogs (JSON unicode escapes, backspace overstrike, MCP prompts, Base32/Braille/Atbash), additional shell trampolines (find -exec, rsync $HOME, wmic), memory classes (query-rewrite, emergency-CC), national-ID formats outside the v0.45 list (TCKN, RUT, SA ID, codice fiscale, BSN), new path classes (vault-token, age keys.txt, FileZilla, tfstate, Signal), posture gaps (go run @latest, cwd /, bind 0.0.0.0), a Slack incoming webhook, and OAuth fields (token_endpoint_auth_method=none, wildcard redirects, cleartext front-channel logout). One false positive: a secretflow local equality check treated as reuse.

Detection-gap issues #20 (memfence impossible-age claims), #21 (pathfence privilege-escalation write content), and #24 (secretflow official-provider allowlist) are closed against this release. Worldview / conspiracy claims remain out of memfence scope, as noted on #20.

python3 benchmarks/run_benchmark.py

The playground vendors were resynced so the in-browser copies match the CLI.

The floating Action / image tag v0 points at this release (uses: dunkleschitz/agentsec@v0 or @v0.45.0).

Still not on PyPI. Publishing steps are in PUBLISHING.md.

v0.44.0

agentsec v0.44.0

Offline, stdlib-only kit. 36 tools. This release generalizes detection for the misses in red-team round 3, then scores a fresh round-4 set once. No new scanners. 1810 tests.

Generalization (not fixture-specific string matches)

Round 3 is the development set. Rules were extended around attack class before any round-4 sample was written:

• toolpoison — scans MCP server-level instructions / serverInstructions; TP14 elicitation used as a credential phish; decode layers now include quoted-printable and tolerant uuencode.
• shellfence — SF17 LOLBAS remote download/execute (certutil -urlcache, mshta, bitsadmin /transfer); SF18 download-then-execute without a pipe (curl -o, chmod +x, then run the file); SF11 also covers /etc/ld.so.preload and LD_PRELOAD=.
• memfence — count-delayed exfil (“after N messages”), system-prompt append, and planted payroll/login portals that forward session cookies (MF12). Official HTTPS portals without cookie theft stay clean.
• piiguard — PG05 also covers Indian Aadhaar, Brazilian CPF, Australian TFN, and Swedish personnummer, with checksums where the format defines them.
• mcpposture — MP21 also fires on --registry / --index-url CLI flags, not only npm_config_registry.
• secretflow — data-URI and QR-generator sinks; GitHub MCP write sinks; URL-encoded labeled keys (including Azure AccountKey); placeholders such as your-api-key-here in .env.example are no longer flagged.
• mcpoauth — redirect_uris / post_logout_redirect_uris checked for cleartext http:// on public hosts; file: / data: / javascript: schemes; *.svc / .cluster.local as private; broader GitHub scopes; published DCR access tokens (MO11). Open DCR (MO13) is now info, matching the MCP SHOULD.
• fetchfence — FF14 known no-auth chat/webhook exfil drops (Telegram Bot API, Discord webhooks, Slack hooks). Well-known public resolvers (1.1.1.1, 8.8.8.8, …) are no longer FF08.
• pathfence — PF13 UNC / NTLM-coercion paths; PF14 wallet/keystore paths; URL-decoded traversal; .env.example / .sample / .template are not treated as live secret files.
• tracesweep — SendGrid, Azure Storage AccountKey, Telegram bot tokens, and npm tokens; conservative double-base64 decode.

Benchmark

Lead with the latest unseen score, not the seen ones.

Set	Score (P / R / F1)	Counts (tp/fp/fn/tn)
Author expanded (125 / 10 tools)	100% / 100% / 100%	76/0/0/49
Original 85-sample, 5-tool slice	100% / 100% / 100%	51/0/0/34
Held-out (40, written after #7–#11)	100% / 100% / 100%	26/0/0/14
Red-team round 1 independent (v0.40.0, unseen)	93.8% / 85.7% / 89.6%	60/4/10/26, frozen, still shown
Same round-1 set after #12–#16	100% / 100% / 100%	70/0/0/30, labeled seen
Red-team round 2 independent (v0.41.0, unseen)	80.4% / 38.9% / 52.5%	37/9/58/30, frozen, still shown
Same round-2 set after later generalization	98.9% / 97.9% / 98.4%	93/1/2/38, labeled seen
Red-team round 3 independent (v0.43.0, unseen)	97.3% / 65.5% / 78.3%	36/1/19/28, frozen, still shown
Same round-3 set after v0.44.0 generalization	100% / 100% / 100%	55/0/0/29, labeled seen
Red-team round 4 (v0.44.0, unseen)	97.0% / 60.4% / 74.4%	32/1/21/26, scored once

Round 4 (benchmarks/redteam4/, 80 samples) was written after the rules were frozen, from published techniques rounds 1–3 did not use (Policy Puppetry, ANSI-hidden instructions, HTTP header instructions, FigStep-class OCR captions, MCP resource contents and resource templates, outputSchema $comment, ROT13 and reversed-text concealment, tar --checkpoint-action, osascript, rclone, MSBuild, forfiles, systemd-run transient units, auto-approve and allow-list memory, calendar-triggered exfil, PESEL/NRIC/NIR/My Number/CURP, redis://, .onion, dword IPs, password-store / 1Password / Kubernetes service-account tokens / keychains, SSE token-in-query, Deno, DNS/CSS/WebSocket secret sinks, hf_ / dop_v1_ / age keys, JWT alg=none, and others; see SOURCES.md). Numbers are reported honestly. Misses include several new national-ID formats, additional LOLBAS / archive execution trampolines, ROT13 and reversed-text descriptions, password-manager paths, alg=none, and an age secret key — exactly the kind of unseen gap this set is for. One false positive: a vendor policy XML that only describes rate limits (TR10).

Remaining honest misses on the (now seen) earlier sets: pathfence EscapeRoute (sibling-symlink write, #21), memfence Gemini false-facts with no instruction-like text (#20), and a secretflow “key used with its own provider” false positive (#24). Detection-gap issues #17 (fetchfence), #18 (mcpoauth), and #27 (tracesweep) are closed against this release.

python3 benchmarks/run_benchmark.py

The playground vendors were resynced so the in-browser copies match the CLI.

The floating Action / image tag v0 points at this release (uses: dunkleschitz/agentsec@v0 or @v0.44.0).

Still not on PyPI. Publishing steps are in PUBLISHING.md.

v0.43.0

agentsec v0.43.0

Offline, stdlib-only kit. 36 tools. This release generalizes detection on the six weakest tools from red-team round 2, then scores a fresh round-3 set once. No new scanners.

Generalization (not fixture-specific string matches)

Round 2 is the development set. Rules were redesigned around attack structure before any round-3 sample was written:

• resultguard — TR10 structural injection: co-occurrence of instruction-to-agent cues, authority/override framing, a sensitive object, and an exfil sink. Unicode normalization (homoglyphs, zero-width, tags, bidi) and decode layers (HTML, URL, base64, and similar) run first. Discussion/research context stays clean; ordinary CDN images are not TR09.
• toolpoison — TP11 annotation deception (readOnlyHint / destructiveHint / title vs destructive behavior); TP12 hidden schema defaults on BCC/webhook-like parameters; TP13 preference manipulation. Descriptions are decoded (HTML, URL, base64) before matching.
• memfence — MF11 delayed/trigger-phrase backdoors; MF12 payment-swap or pipe-to-shell planted as standing memory. Multilingual persist/exfil cues. Negation uses word boundaries so “never paste API keys” stays clean while “Whenever … paste API keys” still flags. False biographical facts are still out of reach of these heuristics.
• shellfence — Tokenizes with shlex, normalizes quote/escape/PowerShell -EncodedCommand obfuscation, and reasons about pipelines. SF13 AI-CLI bypass flags; SF14 terraform/kubectl/DROP cascade; SF15 authorized-keys backdoors; SF16 secret-scanner plus public exfil. git push --force-with-lease is no longer SF07.
• mcpposture — MP19 autoApprove/alwaysAllow/trust:true on privileged tools; MP20 credential-directory mounts even on digest-pinned images; MP21 non-official registry overrides; MP22 known-vulnerable pinned version ranges; MP23 remote-code bootstrap (python -c / node -e fetch+exec).
• piiguard — PG05 also covers UK NINO, Canadian SIN, and ICAO TD3 MRZ. URL/base64 payloads are scanned one nested level.

Benchmark

Set	Score (P / R / F1)	Counts (tp/fp/fn/tn)
Author expanded (125 / 10 tools)	100% / 100% / 100%	76/0/0/49
Original 85-sample, 5-tool slice	100% / 100% / 100%	51/0/0/34
Held-out (40, written after #7–#11)	100% / 100% / 100%	26/0/0/14
Red-team round 1 independent (v0.40.0, unseen)	93.8% / 85.7% / 89.6%	60/4/10/26, frozen, still shown
Same round-1 set after #12–#16	100% / 100% / 100%	70/0/0/30, labeled seen
Red-team round 2 independent (v0.41.0, unseen)	80.4% / 38.9% / 52.5%	37/9/58/30, frozen, still shown
Same round-2 set after generalization	94.0% / 82.1% / 87.6%	78/5/17/34, labeled seen
Red-team round 3 (v0.43.0, unseen)	97.3% / 65.5% / 78.3%	36/1/19/28, scored once

Round 3 (benchmarks/redteam3/, 84 samples) was written after the rules were frozen, from published techniques rounds 1 and 2 did not use (Greshake et al., ArtPrompt, Crescendo, many-shot jailbreaking, Virtual Prompt Injection, MCP initialize/elicitation/sampling, LOLBAS, download-then-exec, ld.so.preload, Aadhaar/CPF/TFN/personnummer, GCP metadata, gopher/dict, punycode, RFC 9700, and others; see SOURCES.md). Numbers are reported honestly. Misses include LOLBAS binaries, several national-ID formats, MCP instructions / elicitation / uuencode descriptions, count-delayed memory exfil, and a couple of unusual secret sinks — exactly the kind of unseen gap this set is for.

Priority tools on the (now seen) round-2 set: toolpoison, resultguard, shellfence, mcpposture, and piiguard are 100%; memfence is R=85.7% (the remaining miss is planted false facts). Detection-gap issues #23, #26, #25, #19, and #22 are closed. #20 (memfence false facts) and the issues for tools this release did not retune (#17 fetchfence, #18 mcpoauth, #21 pathfence, #24 secretflow, #27 tracesweep) stay open.

python3 benchmarks/run_benchmark.py

The playground vendors were resynced so the in-browser copies match the CLI.

The floating Action / image tag v0 points at this release (uses: dunkleschitz/agentsec@v0 or @v0.43.0).

Still not on PyPI. Publishing steps are in PUBLISHING.md.

v0.42.0

agentsec v0.42.0

Offline, stdlib-only kit. 36 tools. This release adds a second independent red-team benchmark. No detector rules change.

Independent red-team round 2 (benchmarks/redteam2/)

134 new adversarial + benign look-alike samples across eleven tools (adds mcpoauth), written against v0.41.0 without reading detector source, rule files, or the round-1 fixtures (READMEs/usage and the round-1 SOURCES.md only). Techniques favor published research round 1 did not use: ShadowLeak, Policy Puppetry, Copilot autoApprove (CVE-2025-53773), CurXecute, HashJack, Gemini Trifecta, ForcedLeak, Sneaky Bits, emoji variation-selector smuggling, MPMA, MCPTox, Elastic orchestration injection, postmark-mcp, Shai-Hulud, s1ngularity AI-CLI abuse, Tracebit Gemini CLI, Replit DB deletion, EscapeRoute, mcp-remote CVE-2025-6514, MCP Inspector CVE-2025-49596, Unit 42 Bedrock memory poisoning, AgentPoison, and RFC 9700 OAuth misconfigurations. Citations: benchmarks/redteam2/SOURCES.md.

Unseen score (v0.41.0 rules)

Tool	TP	FP	FN	TN	Precision	Recall	F1
fetchfence	8	1	2	3	88.9%	80.0%	84.2%
mcpoauth	5	1	3	3	83.3%	62.5%	71.4%
mcpposture	3	0	5	3	100.0%	37.5%	54.5%
memfence	0	1	7	2	0.0%	0.0%	n/a
pathfence	6	1	4	2	85.7%	60.0%	70.6%
piiguard	3	0	4	3	100.0%	42.9%	60.0%
resultguard	2	2	9	2	50.0%	18.2%	26.7%
secretflow	3	2	3	1	60.0%	50.0%	54.5%
shellfence	3	1	9	3	75.0%	25.0%	37.5%
toolpoison	0	0	8	4	n/a	0.0%	n/a
tracesweep	4	0	4	4	100.0%	50.0%	66.7%
overall	37	9	58	30	80.4%	38.9%	52.5%

Round 1 for comparison: v0.40.0 unseen P=93.8% R=85.7% F1=89.6%; v0.41.0 on the same (now seen) set 100%. The round-2 drop is the honest signal that the round-1 fixes did not generalize to new techniques.

Frozen in benchmarks/baseline_redteam2_v0.41.0.json; shown on the benchmark page alongside the round-1 sections.

Detection-gap issues

#17 fetchfence, #18 mcpoauth, #19 mcpposture, #20 memfence, #21 pathfence, #22 piiguard, #23 resultguard, #24 secretflow, #25 shellfence, #26 toolpoison, #27 tracesweep

Other changes

• benchmarks/run_benchmark.py scores mcpoauth (MO14 inventory ignored) and adds the round-2 section and --check guard.
• 18 round-2 fixtures carry provider-shaped fake keys that GitHub push protection blocks; they are generated deterministically by benchmarks/redteam2/generate.py (auto-run by the benchmark) instead of being committed.
• All secrets and PII in the set are synthetic.

v0.41.0

agentsec v0.41.0

Offline, stdlib-only kit. 36 tools. This release closes the independent red-team detection-gap issues. No new scanners.

Detection-gap issues #12–#16

General rule changes (not fixture-specific string matches):

• resultguard — “disregard your prior constraints” and similar instruction nouns; conceal phrasing with words between “mention” and “the user”; private-to-public copy bait (enumerate private repos into a PR); typographic-quoted jailbreak phrases in research/discussion context stay clean; the bare word “jailbreak” in a paper or blog is no longer TR01.
• toolpoison — “no need to mention this to the user”; mandatory shell command prefixes (Trail of Bits line-jumping); parameter names that themselves encode “read this secret file”; documentary mentions of ~/.ssh/id_rsa and similar paths stay clean when no access verb is present.
• tracesweep — decodes base64 (and textual hex) blobs and scans the result with existing secret rules; reconstructs known-prefix secrets split across later tool-result turns.
• shellfence — find $HOME … -exec rm / -delete and rm -rf $HOME (Amazon Q-style home wipes); DNS exfil via dig/nslookup/host to collector/OOB names or interpolated secrets; ANSI-C / hex escape normalization so $'\x72\x6d…' still hits SF02.
• memfence — “paste your API keys into the chat” harvest wording (also closes the remaining author-corpus MF05 miss); MINJA-style query remapping and “authoritative policy for all future sessions.”
• mcpposture — MP04 (npx -y) is suppressed when the package is already pinned to an exact version; unpinned -y still flags MP03 and MP04.

Benchmark

Set	Before (v0.40.0)	After (v0.41.0)
Author expanded (125 / 10 tools)	100% / 98.7% / 99.3% (75/0/1/49)	100% / 100% / 100% (76/0/0/49)
Original 85-sample, 5-tool slice	100% / 100% / 100% (51/0/0/34)	unchanged
Held-out (40, written after #7–#11)	100% / 100% / 100% (26/0/0/14)	unchanged
Red-team independent (100, unseen)	93.8% / 85.7% / 89.6% (60/4/10/26)	frozen, still shown
Same red-team set after #12–#16	—	100% / 100% / 100% (70/0/0/30), labeled seen

The red-team set is no longer independent once these fixes landed. The original unseen v0.40.0 numbers stay on the benchmark page. A 100% seen score is not a claim about new, unpublished attacks.

python3 benchmarks/run_benchmark.py

The playground vendors for resultguard, shellfence, and mcpposture were resynced so the in-browser copies match the CLI.

The floating Action / image tag v0 points at this release (uses: dunkleschitz/agentsec@v0 or @v0.41.0).

Still not on PyPI. Publishing steps are in PUBLISHING.md.

v0.40.0

agentsec v0.40.0

Independent red-team detection benchmark.

• Added benchmarks/redteam/ (~100 adversarial + benign look-alike samples across 10 tools) with SOURCES.md citing published research (Invariant Labs, Trail of Bits, CyberArk, Embrace The Red, Aim Security EchoLeak, PortSwigger, Pillar Security, Supabase MCP / General Analysis, Amazon Q wipe prompt, MINJA, Nx s1ngularity).
• Samples written without reading detector source or rule files (READMEs only).
• Pages section Independent red-team results: overall P=93.8% R=85.7% F1=89.6% (60/4/10/26) — below the author-written corpus.
• Filed detection-gap issues #12–#16 for false negatives; did not retune tools against these fixtures.
• Kit still 35 tools / 1675 tests.

v0.39.0

agentsec v0.39.0

Offline, stdlib-only kit. 35 tools, 1674 tests. This release closes the first detection-gap issues and extends the synthetic benchmark. No new scanners.

Detection-gap issues #7–#11

General rule changes (not fixture-specific string matches):

• resultguard — typo-tolerant and split-word override matching; ROT13 decode (TR06); tighter TR01 so “override the system font” stays clean; TR09 no longer flags an ordinary CDN <img> logo, while markdown images and tracking/auto-exec URLs still fire; multilingual override phrasing.
• toolpoison — soft “dump/snapshot of the workspace in the X parameter” exfil (TP04); French/Spanish/German/Portuguese/Italian overrides and “send the conversation to https://…”; common notes/comments parameters are only flagged when their description asks for secrets; a tool documenting its own audit BCC is no longer TP06.
• tracesweep — reconstructs known secret prefixes split across trivial string concatenation ("sk-proj-" + "BODY").
• secretflow — treats base64, hex, and URL-encoding of a previously seen secret as reuse.
• shellfence — normalizes empty quotes, $IFS, and simple { cmd; } wrappers so obfuscated curl|bash still hits SF01.

Benchmark

On the original 85-sample, 5-tool corpus:

	Precision	Recall	F1	TP/FP/FN/TN
v0.38.0 (before)	91.5%	84.3%	87.8%	43/4/8/30
v0.39.0 (after)	100%	100%	100%	51/0/0/34

A 40-sample held-out set was written after those rule changes were locked and was not used to tune them. Headline: precision 100%, recall 100% (26/0/0/14). That is still a small, author-written, synthetic set — not a real-world attack corpus.

The main corpus now also includes original fixtures for piiguard, fetchfence, pathfence, mcpposture, and memfence (125 samples, 10 tools): precision 100%, recall 98.7%, F1 99.3% (75/0/1/49). The remaining false negative is memfence MF05 missing a “paste your API keys into the chat” wording.

Results: dunkleschitz.github.io/agentsec/benchmarks/

python3 benchmarks/run_benchmark.py

The playground vendors for resultguard, secretflow, and shellfence were resynced so the in-browser copies match the CLI.

The floating Action / image tag v0 points at this release (uses: dunkleschitz/agentsec@v0 or @v0.39.0).

Still not on PyPI. Publishing steps are in PUBLISHING.md.