Evals and Live Results

Evals and Live Results

Two suites under evals/, one runner, graded by hidden tests the agent never sees. The Fake provider makes every suite runnable offline at real-model prices; live runs use the Anthropic profile under a cost cap. History of nightly live runs: docs/EVAL_HISTORY.md.

uv run python -m evals.runner --suite regression --provider fake --trials 5 --min-pass-pow-k 1.0
uv run python -m evals.runner --suite regression --config evals/configs/sonnet_opus_advisor.yaml --parallel 3 --max-usd 6
uv run python scripts/nightly_live.py --max-usd 12          # both suites, appends a row to EVAL_HISTORY.md
uv run python -m evals.tune --provider fake --trials 2      # escalation threshold sweep (ADR 0006)

Regression tasks

Each task is a fixture repository, a Markdown spec and hidden tests; a TaskScript drives it deterministically on the Fake. All ten pass at pass^5 = 1.00 offline.

Task	Tier	What it exercises	Live
t001 add function	T0	the happy path	5/5, about $0.11 per trial
t002 fix failing test	T0 or T1	a bug fix with existing tests	pass
t003 update docs	T1	a docs task the planner must make provable by tests	pass, about $0.37
t004 two features	T1	two plan steps, one commit each	pass
t005 CLI JSON flag	T1 or T2	a feature across module and CLI	pass
t006 multi-file remove	T1	store module, CLI and both test files	pass, $0.35
t007 red CI	T1	the fixture's own CI is red on ruff and pytest; hidden tests run the repo's rules	pass, $0.40
t008 snapshot coupling	T1, pinned	a change that breaks a snapshot test and a README test outside the filtered run	pass, $0.39
t009 verify-only lint	T1, pinned	an undefined name in a module no test imports: only verify's lint sees it; first live debugger run	pass, $0.34, debugger $0.06
t010 wide rename	T1, pinned	a rename across nine modules; the advisor reviews the diff before commit (ADR 0008); first live advisor runs	pass, $0.64, two advisor calls

Capability scenarios

c001 repeated error resolved by the advisor, c002 plan drift, c003 advisor cap. They inject failure modes into the Fake coder and are Fake-only; they are the control-flow evidence behind the escalation thresholds.

What the live runs taught

• A scripted fake reproduces the shapes you imagined; the first live runs found five harness defects the Fake structurally could not (node ids in -k expressions, a directory passed to view_file, an over-conservative budget pre-check, empty tests_added on bug fixes, budgets sized for the hinted tier).
• Keep the working directory of the first live run of every task and read the artifacts and the per-role cost ledger before changing anything.
• A competent coder makes verify pass, so forcing a graph path needs a failure outside the worker's tools (t009) or a design change (ADR 0008 for the advisor).
• Live T1 runs cost three to five times a T0 run on the same task; transcript growth per coder turn is the cost driver.

Nightly

scripts/nightly_live.py runs regression and capability on the Anthropic profile under a total cap and appends one row per suite to EVAL_HISTORY.md; it is registered on the dev machine as a Windows scheduled task. The GitHub nightly workflow runs the same commands and switches to the live provider only when a repository secret exists.