You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Changed
E2B is no longer a planned live tier: no account (I-029, wontfix). Local backend for own fixtures, Docker for untrusted repositories (ADR 0003).
Added
t008_snapshot_coupling: a formatting change that breaks a CLI snapshot test and a README example test outside the coder's filtered run; the first regression task built to make verify's full-suite step fail and the debugger run live.
t009_verify_only_lint (T1, pinned): an undefined name in a module no test imports; the coder's run_tests is green and only verify's lint step fails, so the debugger must run and wire the constant rather than delete the module (hidden tests check both). First live debugger run: one turn, $0.06 of a $0.34 trial, fix plus sweep. Fake scripts gained Step.debug_edits for the debugger's turn.
tier_pin: on an eval task runs that tier without consulting the router, which rated t008 T0 and skipped the nodes the task exists to exercise (I-050). RunContext.tier_pin; route records classifier_model="pinned".
Two realistic regression tasks: t006 (a change spanning the store module, the CLI and their tests) and t007 (a fixture whose own CI is red on ruff with the repo config and on pytest); hidden tests run the fixture's own lint rules. One live trial each passed ($0.35 and $0.40).
Suite cost cap: evals/runner --max-usd; the report records spend, cap and skipped tasks.
Nightly live suite: scripts/nightly_live.py (profile auth, cost cap, rows in docs/EVAL_HISTORY.md) registered as a Windows scheduled task on the dev machine with --max-usd 12; the GitHub nightly workflow gained the cap, parallel trials and auth-token secret support.
scio report tool signals, counted from the worker transcripts with no model call: calls and error rate per tool, repeated identical calls within a turn, view_file re-reads, and whether the documenter edited docs when code changed. The documenter's transcripts are now kept as log artifacts like the coder's.
Fixed
Eval trials leaked their temp directory on Windows (636 directories): read-only git objects and an open effect-log handle, both hidden by rmtree(ignore_errors=True). remove_tree clears the bit, RunContext.close() releases the handle (I-049).
Eval fixtures are copied without __pycache__ and tool caches, so bytecode left by running a fixture's tests in place can no longer become tracked content that shows up in the agent's commits (I-048).