v0.2.0 - 160x fewer false positives, reproducible scoring

Precision 1.000, Recall 0.864, F1 0.927 against a public ground truth, and 23 findings across 285,463 lines of clean code, down from 3,642.

Precision, recall and F1

demo/ground-truth.json documents 22 planted vulnerabilities across all four taxonomy layers. demo/safe is the same code with every one fixed, so it contributes only true negatives and precision is measured directly rather than sampled.

python benchmark/score_demo.py

	TP	FP	FN	Precision	Recall	F1
All layers	19	0	3	1.000	0.864	0.927
LLM	1	0	0	1.000	1.000	1.000
Traditional web	5	0	0	1.000	1.000	1.000
MCP	11	0	1	1.000	0.917	0.957
Agentic AI	2	0	2	1.000	0.500	0.667

The three misses are all properties of protocol state rather than of any line of code, which is why the Agentic AI layer scores lowest. These fixtures carry one clean instance of each class, so read this as evidence that the rules fire and discriminate, not as an estimate of recall on unseen code.

On 285,463 lines of clean, officially maintained MCP code, findings went from 3,642 to 23. A 160x reduction, and the one finding above the reporting threshold is a genuine bug in the official SDK.

Try it in thirty seconds

The repository ships the same MCP server twice, with every vulnerability in the first one fixed in the second.

pip install -e "mcp-scan[all]"
mcpvuln demo/vulnerable      # 20 findings, 12 categories, all four layers
mcpvuln demo/safe            # 0 findings

No API key. No network. The second command is the one that matters: any scanner finds planted bugs, but one that also fires on the corrected version is not measuring anything.

What changed

Detection

• Fixed a regular expression responsible for 2,061 of 3,642 findings. wget .* | sh used an unescaped |, which in a regular expression is alternation rather than a shell pipe, so the alternative ' sh' matched the word "should".
• Fixed four duplicate dictionary keys that silently discarded four pattern lists: 33 keys written, 29 surviving, 11 expressions never executed.
• Removed twenty unanchored bare-substring patterns (latest, http://, open(, while True) that fired on comments, documentation and URLs.
• Made case-insensitivity opt-in per pattern. Applied globally, DES\s*\( matched includes(.
• Matching is whole-file rather than line-by-line, and comments, docstrings and prose are suppressed.
• Every finding carries a confidence score used for ranking, filtering and the informational threshold.
• Added suppression directives: # mcpvuln: ignore-file, # mcpvuln: ignore, # mcpvuln: ignore[rule.id].

Scoring is now computed, not generated

• CVSS v4.0 base scores come from the cvss implementation of the FIRST specification. The same finding scores identically on every run; previously a language model was asked to write a score in prose.
• SSVC is evaluated as the published decision tree. Exploitation is never reported as active, because a source scanner observes code, not exploitation in the wild.

Architecture

• New scan contract: a versioned JSON document that is the only thing crossing a stage boundary. Save it with --json, diff it across commits, score it offline.
• SecurityAnalysisTeam is now SecurityAnalysisPipeline. The previous class presented the stages as a coordinating multi-agent team, but the coordinating model was constructed and never invoked, and OPENAI_API_KEY appeared nowhere in the repository. The old name remains as a deprecated alias.
• Detection requires no API key and no network. Only --narrative and --threat-intel reach out, and both degrade to a warning without their key.
• Removed findings[:20], which silently discarded 99.4% of findings on a repository of any size.

Testing

• 140 tests, from zero. Every defect above is held down by a regression test.
• CI on Linux and Windows across Python 3.9, 3.11 and 3.12, plus a benchmark job that fails if the false-positive rate exceeds 0.05 per 100 LOC.
• New benchmark with a pinned benign corpus, so precision is measurable rather than assumed.
• Fixed an install broken since the first release: setup.py read README.md from the working directory rather than relative to itself, so the documented install failed on a fresh clone.

Benchmark

Version	Findings on 285,463 clean lines	Per 100 LOC
v0.1.0	3,642	1.28
v0.2.0	23	0.008

Corpus: modelcontextprotocol/python-sdk at d060b36 and modelcontextprotocol/servers at d73f99e. Reproduce with python benchmark/run_benchmark.py --fetch.

Upgrading from 0.1.0

SecurityAnalysisTeam still works but warns. Reports are written with --out rather than to the working directory. Full notes in CHANGELOG.md.