codesecbench-truth

Ground-truth labels for the CodeSecBench Tier C app-shaped benchmark corpus.

What's in this repo

targets/
  cst-nextjs-chat.json          # truth for github.com/getdebug-ai/cst-nextjs-chat
  cst-vite-rag.json
  cst-sveltekit-stream.json
  cst-express-agent.json
  cst-fastapi-tools.json
  cst-crewai-multiagent.json
schema.json                     # JSON Schema for the truth files
score.js                        # minimal vendor-side scorer (~80 lines, zero deps)
CHANGELOG.md                    # versioned label revisions

Each targets/*.json is a list of trueState rows — one per file:line that exercises a security pattern. Every row carries:

• file, line — the exact location in the corresponding target repo
• category — one of client-side-llm-key, prompt-injection, unsafe-tool-output, unsafe-role-merge, pii-in-prompt, unbounded-stream
• severity — critical | high | medium | info
• trueState — vulnerable (a real finding) or safe (a near-miss that detectors commonly mis-flag)
• rationale — a paragraph explaining the adjudication, with CVE/CWE references where applicable
• provenance — human-adjudicated | oracle-confirmed | fix-derived
• confidence — 0 (threat-model-dependent), 1 (single reviewer), 2 (oracle-confirmed or multi-rater)

Total v0.1 corpus: 6 repos × ~24 rows each ≈ 144 labeled lines spanning all 6 AI-app security categories. ~72 of them vulnerable, ~36 safe near-misses, ~36 borderline cases (the rationale strings on these explicitly mark them as borderline).

The "don't peek" norm

CodeSecBench works the same way every public security benchmark works (OWASP Juice Shop, NodeGoat, SQLi-Labs, OWASP Benchmark): the corpus is public, the truth is public, and the social contract is don't pre-train your detector on the truth file.

We can't enforce this. We document it. If your tool's recall on CodeSecBench is suspiciously perfect — especially on the borderline rows where reasonable engineers disagree — the community will notice and contest it through the PR process. Same review pressure that keeps any open benchmark honest.

If you publish a number against CodeSecBench, cite the truth version you measured against (see CHANGELOG.md). A number quoted against 0.1.0 is not directly comparable to a number quoted against 0.2.0 if labels were revised in between.

How to run your tool against the corpus

# Clone all 6 targets (or whichever subset you support — Python tools can skip
# the JS/TS-only ones, JS/TS tools can skip the Python-only ones).
for repo in cst-nextjs-chat cst-vite-rag cst-sveltekit-stream \
            cst-express-agent cst-fastapi-tools cst-crewai-multiagent; do
  git clone "https://github.com/getdebug-ai/$repo" "/tmp/$repo"
done

# Run YOUR tool over each repo, capture findings as a JSON array of
# { file, line, category, severity? } objects.
your-tool scan /tmp/cst-nextjs-chat --json > /tmp/cst-nextjs-chat.findings.json
# ... repeat for the other 5 repos

# Clone this truth repo (or just download the targets/*.json files raw).
git clone https://github.com/getdebug-ai/codesecbench-truth
cd codesecbench-truth

# Score:
node score.js \
  --findings /tmp/cst-nextjs-chat.findings.json \
  --truth targets/cst-nextjs-chat.json

The scorer outputs per-category TP / FP / FN / precision / recall, plus a per-category breakdown of which borderline rows your tool agreed/disagreed with.

Adding your tool's results to the public leaderboard

Open a PR adding results/<your-tool>-<version>-<YYYY-MM-DD>.json with:

{
  "tool": "your-tool",
  "version": "1.2.3",
  "ranAt": "2026-06-15T12:00:00Z",
  "truthVersion": "0.1.0",
  "perCorpus": [
    { "target": "cst-nextjs-chat", "tp": 10, "fp": 3, "fn": 2, "precisionByCategory": { ... } },
    ...
  ],
  "rawFindings": "https://gist.github.com/...your-published-raw-output..."
}

The leaderboard rendering lives in docs/leaderboard.md and is rebuilt from results/ on every merge.

Disputing a label

Truth labels are not infallible — especially the confidence: 0 borderline cases. To dispute:

1. Open a PR against targets/<repo>.json with the proposed trueState change.
2. Cite: a CVE, a public advisory, a working PoC, or a published threat-model framework that supports the new label.
3. Reviewers (the current maintainer + any external co-maintainers — see GOVERNANCE.md) discuss in the PR.
4. Accepted disputes bump truthVersion and add a CHANGELOG entry.

Why the truth repo is separate from the target repos

So SAST scanners that crawl the target source tree never see the labels. This is the structural reason for the split. The pointer from each target back to this repo lives in BENCHMARK.md at the target's root — humans find it; scanners don't follow external URLs.

License

MIT.