develop metric for vulns per mtok and/or vulns per CPU-time (or vulns per some combination of mtok and time)

[quinn-dougherty]

register metric to inspect-ai (or pydantic-ai)

[quinn-dougherty]

What the metric should capture (and what it should not)

Writing up the design reasoning before/alongside the implementation.

The thing we're measuring

Incidence of vulns per unit of resource spent, as a function of (model, agent architecture) — the repo's "core crux". A fuzzing run burns two genuinely different resources:

• tokens — LLM inference (analyze source, write/repair harnesses, triage). Priced in $/Mtok by the provider.
• fuzz-time — wall-clock the fuzzer burns executing the target after a harness compiles. Independent of the model; priced in $/CPU-hour by the cloud.

Why vulns / (mtok * time) is the wrong shape

A product denominator implies tokens and time mutually amplify each other, which is economically meaningless. Three independent ways to see it:

1. No price exists for the product. vulns / (token-hours) would need a $/(token·hour) price. That doesn't exist. But $/Mtok and $/hour both exist, and their linear combination lands in dollars — a unit with a real referent (your invoice). You get two invoices (API + cloud); your spend is their sum, never their product.
2. Homogeneity. An efficiency ratio should be homogeneous of degree -1 in the resource vector: a config that does the same job at 2x scale (ran 2x longer, thought 2x harder, found the same bugs) should score 2x worse — you paid double for nothing. A linear cost gives exactly that. The product T*W is degree -2: it scores you 4x worse. That square is the bug.
3. Boundary blowup. With V/(T*W), starving either resource sends the metric to infinity (write one cheap harness, fuzz forever → "infinitely efficient"). A linear cost never blows up.

"Are there exponents?" No. Generalizing to a*T^p + b*W^q: unit prices are constant (the 10-millionth token costs the same as the first), so the cost is linear → p = q = 1. Requiring degree-(-1) homogeneity pins both exponents to 1. The genuine nonlinearity (diminishing returns / coverage saturation) lives in the production function V(T,W) — that's the signal we're trying to measure, not something to bake into the denominator.

The asymmetry: token->$ is informative, CPU->$ is arbitrary

Converting tokens to dollars is an informative normalization — the token price covaries with the model (Opus tokens cost more because they're worth more), so "a dollar is a model-size-class-adjusted token." That conversion carries the capability signal.

Converting CPU-time to dollars is not informative — a CPU-second is fungible commodity compute, and its cloud price is an exogenous constant that doesn't covary with the thing being evaluated. Dropping it into the denominator just arbitrarily reweights the two terms toward whatever's cheaper at today's spot price.

Resolution: charge fuzz-time in tokens via the "kappa opportunity cost"

Keep everything in the currency that carries the capability signal — tokens — and charge fuzz-time at its opportunity cost in tokens. The agent is idle while the fuzzer runs, so a fuzz-second costs the tokens the agent could have generated in that second:

kappa = total_tokens / model_thinking_seconds        # tokens/sec, MEASURED from the run
D     = total_tokens + kappa * fuzz_seconds           # token-equivalents

kappa is endogenous — measured from each run (sum of model-event working time) — so no external price ever enters. Bigger/slower models have smaller kappa, so their fuzz-seconds are charged fewer tokens: auto-adjusted by the model's own economics.

Consistency check: D == kappa * (thinking_seconds + fuzz_seconds) == kappa * total_walltime. So this is just vulns-per-walltime (the core crux) expressed in the token currency, with the model's own rate as the scale. Dollars only ever enter through the one price we trust: $/vuln = D * ($/Mtok) / V.

Pooling, not averaging

Rates are pooled across samples — sum(V) / sum(resource) — never the mean of per-sample ratios. A single sample finding 1 vuln in 0.1 Mtok would otherwise dominate, and zero-resource samples make per-sample ratios undefined. Pooling gives the true incidence rate over the eval.

Implemented metric set (registered to Inspect-AI)

Attached to live_fuzzing_scorer in scorers/efficiency.py:

• vulns_per_mtok — vulns per million tokens (pure token rate)
• vulns_per_fuzz_hour — vulns per fuzzer wall-clock hour (the core-crux rate)
• token_rate — kappa, tokens/sec of model thinking (diagnostic)
• vulns_per_mtok_equiv — headline: vulns per million token-equivalents, fuzz-time charged at the kappa opportunity cost

Expected token-expenditure-per-vuln (the June SoW milestone) is just 1 / vulns_per_mtok_equiv — left as a downstream reciprocal so the (common) zero-vuln case stays a clean 0.0 rate instead of an undefined inverse.

Deliberately out of scope (open for discussion)

• Dollars are left as a downstream multiply by the per-model token price, rather than baked in — keeps any pricing table out of the metric.
• V = unique crash hashes (top-5-frame stack-hash dedup), not CASR clusters or ground-truth-ARVO-vuln dedup. Tightening the numerator's dedup is a separate question.
• W = sum of capped start_fuzzing durations (deterministic, single-core CPU-seconds proxy). Multicore/swarm would need cores * walltime; the swarm scorer already has a separate crashes_per_cpu_hour.