We scanned real AI/agent repos with SecureAI-Scan — here's the evidence (and what it cost to keep zero false positives)

[akanthed]

We scanned real AI/agent repos with SecureAI-Scan — here's the evidence (and what it cost to keep zero false positives)

Most scanners in this space prove their claims on fixtures they wrote themselves. That's necessary but not sufficient — it only proves the scanner behaves on code written to test it. So npm run regression runs the built CLI against a curated set of real, public LLM/MCP/RAG repos and checks the result against a hand-reviewed baseline, gated in CI. Here's what that's actually turned up.

The headline result: a labeled ground-truth test

cisco-ai-defense/skill-scanner ships an evals/ corpus built to evaluate Agent Skill scanners — pre-labeled malicious/ and safe/ directories, so this is a graded test, not a judgment call.

• 6/6 in-scope malicious fixtures fired: invisible Unicode smuggled into a skill body, a skill that downloads and exec()s a remote payload, a skill whose body reads "Ignore all previous instructions..."
• 0 findings on anything labeled safe/
• 0 findings across all 18 real skill bundles in anthropics/skills and all 14 in vercel/ai — no false alarms on skills nobody built to be malicious

What zero-false-positive actually costs — a live example from this week

We just added static config scanning for LiteLLM Proxy (hardcoded secrets, plaintext provider endpoints, missing guardrails). None of the repos already in the regression set exercise those rules, so we added BerriAI/litellm itself — the official repo, ~7,000 TS/JS files plus a large Python codebase, a genuinely harder target than anything we wrote ourselves.

First pass against it surfaced three real bugs before it confirmed one real finding:

1. A rule correctly detected a hardcoded secret existed somewhere in a config, but anchored the finding to the first line containing the key name api_key in the whole file — not the line with the actual offending value. In a file with dozens of api_key: entries, the reported evidence could point straight at an os.environ/... reference and contradict itself.
2. LiteLLM's own docs/tests use placeholder values like fake-key, sk-lar1-demo to demonstrate config shape. Flagged as leaked secrets until we added a "does this look like a random credential blob or a human-typed phrase" check.
3. The most serious one: a single file crashing our TypeScript type-checker (an unrelated large-monorepo edge case) was silently killing the entire scan — not just one rule's findings, all of them. A scan that goes quiet instead of erroring looks identical to "clean." That's worse than any false positive. Fixed by isolating each rule so one crash can't take the rest down.

Two more false positives (a settings description containing the words "system prompt," a URL parser flagged because its parameter happened to be named url, Python's re.search() confused for a vector-store search) surfaced once the scan could actually complete.

After all six fixes: the scan came back with exactly one confirmed-real finding — a plaintext http:// provider endpoint in a proxy config — and every fix shipped with a permanent regression fixture so none of the six can come back silently.

Full writeup with source lines: docs/RealWorldFindings.md

Try it

npx --yes secureai-scan@latest scan .

Offline, no account, no config. Findings default to proven/likely evidence only — heuristic matches are opt-in behind --paranoid.

Feedback, false positives, and "you missed this" reports are genuinely wanted — that's exactly how the six bugs above got found.