Feature Benchmarks and Costs

Feature: Benchmarks & Costs

Two complementary measurement systems: benchmark quality (does the pipeline actually find real bugs?) and observed cost economics (what does it cost to find them?) — plus a real-world triager accept-rate that closes the loop back to actual disclosure outcomes.

Benchmarks (argo bench)

Scores findings quality against a suite of labeled cases (<case>/case.json + expected_findings.json). For each case it runs the pipeline, then matches validated findings to labels: a match requires the normalized CWE to agree (or be in the label's aliases) and the file to match, optionally within a line tolerance. Labels are treated as exhaustive — unmatched reported = false positive, unmatched label = false negative — yielding precision / recall / F1, overall and sliced by archetype and by CWE.

argo bench --suite benchmarks/my-suite --fixes --ab-audit-model claude-sonnet-4-6

• --fixes folds in Remediation & Fixes's patch-verified rate.
• --ab-audit-model runs the suite a second time under a different audit model and reports the precision/recall/F1 delta — the mechanism behind cross-Multi-Backend comparisons.
• --parallel-cases N runs cases concurrently, so corpora can scale.
• The whole harness runs on the mock runner at zero tokens; only --runner headless measures real model quality (and costs money).

Reproducible corpora (commit pin)

A case.json may set "commit": "<sha>" to pin the repo at a specific revision — essential for a URL-backed known-CVE case, otherwise a URL clones the default head where the bug may already be fixed. Argo fetches/checks out exactly that revision (also exposed as argo pipeline --commit). Because ingest clones repo even on the mock runner, URL corpora live under a separate suite (e.g. benchmarks/corpora/) so the default benchmarks/ mock harness stays offline. Bundled real case: benchmarks/corpora/gguf-tools-oob/ (antirez/gguf-tools pinned at the audited commit, labeled with Argo's confirmed heap-OOB / integer-overflow findings, several fixed upstream).

Real-world triager accept-rate

argo feedback records real outcomes for reported findings — whether a human triager actually accepted or rejected them — either one at a time or bulk-imported. argo quality pairs the accept-rate (the human-judged precision proxy) with the latest benchmark recall into quality.json. Neither number alone is the result; the pair is. The private source of truth for this feedback lives outside the public repo (a separate findings registry); Argo only ingests it into the local ledger — no private disclosure data ships in the public tool.

Why the audit model is the tunable that matters

Validation (Adversarial Validation) can only remove false positives — it can never recover a bug the audit model never surfaced in the first place. So the audit model alone sets the missed-bug rate. --calibration forces the audit stage onto the strongest model (effectively all-Opus); this is recommended for the first few programs against a target class, and for any high-value target — drop to a cheaper model only once a target class reliably yields good findings.

Cost analytics (GET /costs)

Observed economics straight from the ledger — real total_cost_usd per call, not a static price list: average cost per run, by-model breakdown ($/call, $/1k output tokens), by-stage breakdown (where the money actually goes), recent-run costs, and the cheapest model per 1k output tokens. Fed back into "Let the AI choose" (see Web UI), whose recommendation appends your observed average run cost to its rationale.

Related

• Design Decisions — why benchmark recall + accept-rate together are the paper's headline result, and the threats to validity they're meant to address.
• Web UI — the Benchmarks and Costs pages surfacing all of this read-only.