malware-test-prs

A controlled testbed for measuring security-scanner accuracy.

This repository contains intentionally vulnerable code. Every file under vulns/ is a deliberately planted, fully documented, non-functional vulnerability pattern that exists so automated scanners can be scored against a known answer key. A GitHub Actions workflow runs the ThreatCrush CLI against every pull request and reports how much of the corpus it caught.

⚠️ Read this before cloning

The vulnerabilities here are intentional. If a scanner flags this repository, it is working correctly — that is the entire point.

Nothing here is a working exploit. Every payload is unreachable dead code, every hostname is an unresolvable reserved domain, every credential is a published example or a dead placeholder, and no dependency can be installed. There is no malware in this repository despite its name.

Do not copy code from vulns/ into a real project. It is broken on purpose.

---

What this is for

Scanner vendors and security teams need to answer two questions that marketing copy does not: what does this tool actually catch, and what does it flag that is perfectly fine. Answering them needs a corpus where the right answer is known in advance. That is this repository.

Each test case pairs a vulnerable implementation with a safe one:

el.innerHTML   = req.query.name;  // VULNERABLE: CWE-79 sink
el.textContent = req.query.name;  // SAFE: textContent does not parse HTML

The safe lines are the control group. A scanner that flags them is pattern-matching on syntax rather than following data flow — and measuring that is most of the value here.

What is in it

Test cases	31
Expected detections	77 (76 marked lines + 1 scored at file level)
False-positive traps	39 SAFE: lines
Languages	JavaScript, Python, Go, Java, Ruby, JSON, dotenv, text
CWE categories	18

Full index: docs/VULNERABILITY_CATALOG.md

vulns/
├── javascript/    XSS, eval injection, SQLi, command injection, SSRF,
│                  prototype pollution, open redirect
├── python/        SQLi, command injection, Jinja2 XSS + SSTI, path traversal,
│                  pickle deserialisation, TOCTOU
├── go/            SQLi, SSRF, command injection, integer overflow
├── java/          SQLi, XSS, ObjectInputStream deserialisation
├── ruby/          SQLi, eval/send RCE, html_safe XSS
├── secrets/       AWS keys, GitHub PATs, Slack webhooks, private key material
└── supply-chain/  Typosquatting, dependency confusion, obfuscated eval,
                   env-var exfiltration disguised as debug logging

How safety is enforced

Five rules, all checked mechanically by scripts/validate-test-case.sh on every pull request:

1. Nothing executes — every payload is behind if (false), if False:, NEVER_RUN, or a //go:build ignore tag.
2. Nothing reaches the network — every host literal uses a domain reserved by RFC 2606 / RFC 6761 (.invalid, .example, example.com), which can never be registered.
3. Nothing is destructive — no disk writes, no process spawning, no deletion, no mining, no self-replication.
4. No live credentials — published vendor examples, structurally valid dead placeholders, or key material generated solely to be committed here.
5. Nothing installs — dependency manifests live only under vulns/supply-chain/, pin versions that exist on no registry, and point at unresolvable registries.

Details and rationale: docs/CONTRIBUTING.md.

Current results

Two scanners measured against the corpus so far:

Scanner	True positives	False positives	Character
ThreatCrush v0.1.0	15.6% (12/77)	0.0% (0/39)	Secrets only — credentials yes, code vulnerabilities no
Semgrep 1.172.0 (p/security-audit, p/secrets)	27.3% (21/77)	7.7% (3/39)	Code patterns yes, secrets shapes largely no

They are close to mirror images, which is the point of keeping a fixed corpus: the numbers are comparable because the answer key does not move.

Two caveats before quoting either figure:

• Dead-code guards depress the true-positive rate. Every payload sits behind if (false), so analyzers that prune unreachable branches under-report. Part of both scores is that, not a genuine detection gap.
• Secrets fixtures are degraded. GitHub push protection rejected the originals, so the Slack fixtures measure pattern detection rather than validated-credential detection.

Both are documented in docs/SCANNER_INTEGRATION.md.

Running it yourself

# Check the corpus against the safety rules
bash scripts/validate-test-case.sh

# Regenerate the catalog from the test-case headers
python3 scripts/generate-catalog.py

# Score any SARIF-emitting scanner against the answer key
semgrep scan --config=p/security-audit --config=p/secrets \
  --sarif --output=results.sarif --no-git-ignore vulns/
python3 scripts/validate-coverage.py \
  --sarif results.sarif --catalog vulns/VULNERABILITY_CATALOG.json

The scoring layer consumes plain SARIF 2.1.0 and is not ThreatCrush-specific. A reference Semgrep baseline, the metric definitions, and an important caveat about dead-code guards suppressing taint analysis are in docs/SCANNER_INTEGRATION.md.

Contributing

New test cases are welcome — especially ones that current scanners miss. A missed detection is the most useful contribution here, because it marks a real gap. See docs/CONTRIBUTING.md for the required metadata and the safety rules.

Scope

This repository is for defensive security research: measuring and improving automated detection. It is not a malware collection, not an exploit toolkit, and not a resource for attacking systems you do not own. Contributions aimed at evading detection in someone else's environment, rather than measuring detection in this one, will be closed.

Security concerns about this repository itself: open an issue, or see docs/CONTRIBUTING.md.

License

MIT — see LICENSE.