vuln-agent-eval

An evaluation harness that measures how well an autonomous LLM agent finds known vulnerability classes in Solidity smart contracts — scored against a ground-truth labeled corpus and compared to static-analysis and single-shot baselines on precision, recall, false-positive rate, and cost per contract.

The design question it answers: when does multi-step agentic structure actually beat a single prompt, and what does it cost? The agent's discovery pass is identical to the single-shot baseline — the only difference is an adversarial verify stage — so any change in the score is attributable to that stage and nothing else.

Headline finding (40-contract pilot): the value of that verify stage is model-dependent — it raised F1 for the Claude models and lowered it for the already-precise DeepSeek models. The result is how a model behaves inside an agent, not a single model's rank. See docs/RESULTS.md.

⚠️ The default make demo run uses a deterministic mock provider: its numbers are ILLUSTRATIVE (no model is queried) and exist so the harness runs end-to-end and on CI with no API key. Real numbers come from a real provider (e.g. make matrix). The static-analysis baseline always runs live when slither is installed.

What's inside

Module	Role
vae.llm	Provider seam (Protocol): mock, claude, deepseek, or your own model — nothing else depends on which
vae.agent	The system under test: discover candidates, adversarially verify each, and (opt-in) judge false positives
vae.eval.systems	The compared systems: agent, single-shot baseline, Slither static baseline
vae.eval	Ground-truth corpus loader + scorer (precision / recall / F1 / false positives)
vae.report	Terminal scoreboard (and the numbers behind it)

Quickstart (no API key)

pip install -e '.[dev]'
make demo     # full eval on the mock provider + sample corpus -> scoreboard
make test     # unit-test the scorer

Example (mock LLM; Slither row is real if installed):

system        Precision  Recall   F1    False positives   Calls/contract
agent             100%     100%   100%          0               2.3
single_shot        50%     100%    67%          2               1.0
slither           100%      50%    67%          0               0.0

Three systems, three distinct failure modes: single-shot over-flags (low precision), Slither is precise but blind to the logic bug (low recall), the agent recovers both by verifying.

Concepts & metrics

• corpus — the labeled test set: a folder of contracts whose vulnerabilities are already known (each .sol has a .json answer key). You can only grade a bug-finder if you know the right answers.
• eval — running the systems over the corpus and grading them against those answer keys. (Corpus = the exam questions + answer key; eval = sitting the exam and scoring it.)
• single_shot — ask the model once ("find bugs"), take the answer as final.
• agent — the same first ask, then a second verify pass that tries to refute each claimed bug and keeps only survivors. The verify step is the single difference, so the score delta is attributable to it.

metric	plain meaning	formula
Precision	of the issues it flagged, the share that were real (fewer false alarms)	TP / (TP + FP)
Precision (judged)	precision after a strict LLM-judge re-checks each false alarm, so a correct-but-unlabeled finding isn't punished (opt-in --judge)	(TP + confirmed) / (TP + FP)
Recall	of the real bugs present, the share it found (fewer misses)	TP / (TP + FN)
F1	one score that's high only when precision and recall are both high	2·P·R / (P + R)
False positives	count of false alarms (flagged, not actually a bug)	—
Calls/contract	average LLM calls per contract (speed/effort)	single_shot = 1; agent = 1 + verifies
Cost/contract	average dollar cost per contract (tokens × price)	—

Shown as percentages. The smoke-detector intuition: precision is "when it beeps, is there really a fire?", recall is "of the real fires, how many did it catch?". Underlying counts: TP = real bug correctly found, FP = false alarm, FN = real bug missed.

Run it live

cp .env.example .env          # set ANTHROPIC_API_KEY and/or DEEPSEEK_API_KEY
pip install -e '.[llm]'
make matrix                   # the enabled models in models.toml, on the 3 samples

Choosing models (how strong a model do you actually need?)

Edit models.toml and flip enabled = true/false per model — no code change. The scoreboard then gains one row per system×model, so you can see whether a cheap model matches an expensive one on precision/recall and at what cost. Defaults on: Claude Sonnet, Claude Haiku, DeepSeek V4 Flash (Opus and DeepSeek Pro are off to save budget). Ad-hoc alternative without the file:

python -m vae eval --models claude:claude-haiku-4-5,deepseek:deepseek-v4-flash

Cheap sample vs. the full benchmark

--per-class N keeps up to N contracts of each vulnerability type (balanced, cheap); omit it for the full set. Easiest via the Makefile:

make eval-sample              # ~2 per class across enabled models  (pennies)
make eval-sample PER_CLASS=5  # a bit larger
make eval-all                 # all 143 SmartBugs contracts          (still < ~$2 on the cheap models)

Every run also writes a self-contained Markdown report (--out, set by these targets) — the scoreboard plus a Findings section listing what each system reported, what the verifier refuted, and what was missed.

Why precision looks low — and the --judge correction

SmartBugs labels one bug per contract, but many contracts contain more. A correct finding on a different real bug is scored as a false positive, so raw precision is a conservative floor. Add --judge: a strict LLM-judge re-checks each false alarm, and findings it confirms as genuine-but-unlabeled stop counting against precision. The scoreboard gains a Precision (judged) column shown next to strict precision — an honest range rather than a single understated number.

python -m vae eval --models deepseek:deepseek-v4-flash --corpus data/corpus/smartbugs \
  --per-class 2 --exclude-class other --judge

Scoring (short version)

A finding is a true positive iff its vuln_class matches a ground-truth label and its line is within --line-tol (default 3) of a labeled line; matching is greedy and one-to-one. Cost is computed from real token counts × current model pricing. Full detail in docs/architecture.md.

Corpus

data/corpus/samples/ holds three hand-labeled contracts that exercise the matcher (committed, used by the keyless demo). For a real benchmark, scripts/ingest_smartbugs.py converts the public SmartBugs-curated dataset (143 labeled contracts) into the same <id>.sol + <id>.json shape — the loader and scorer need no changes:

git clone --depth 1 https://github.com/smartbugs/smartbugs-curated /tmp/smartbugs-curated
make corpus-smartbugs        # -> data/corpus/smartbugs/ (gitignored; regenerate locally)
make eval-sample             # cheap subset across enabled models; or `make eval-all`

See data/corpus/README.md.

Docs

• docs/RESULTS.md — latest scoreboard from a real run (and the mock, labeled).
• docs/MOTIVATION.md — why this measurement matters and the approach taken.
• docs/architecture.md — design, the provider seam, scoring, in depth.
• docs/ROADMAP.md — from scaffold to a scaled, baseline-beating artifact.

Status

Runs end-to-end on the mock provider (CI, keyless) and against Claude and DeepSeek. Multi-model matrix, balanced cheap sampling, a 143-contract SmartBugs corpus, and an opt-in precision-floor LLM-judge are wired up; see docs/RESULTS.md. Next: a cross-model judge, per-class capability curves at full scale, and an attacker↔defender tournament (docs/ROADMAP.md). MIT licensed.