You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
leave a mean gate undetermined when an attempt has no value
skip a measure that is broken on some tasks, and stop when a measure is ready on no task
run takes the config or run directory only as a positional argument. --config, --resume, --jobs, --guidance-name and every evaluation flag (--harness, --model, --reasoning, --guidance, --skill, --plugin, --tasks, --task, --measure, --review-*, --repeats, --seed, --timeout) are removed; set them in the config. run --dry-run prints a summary instead of JSON unless --json is given. The plan's baseline key is now vanilla. examples/evaluation.yaml moved to src/pluginbench/evaluation.example.yaml.
Features
accept per-test numbits sidecars for LCOV (87d8b64)
add pluginbench compare to check gates across saved evaluations (c7031f9)
add status, review and init; take the config only as a positional argument (eee56a0)
cache each finished vanilla attempt as soon as it completes (11c368a)
check every predictable failure before the first paid call (e3ff981)
check every predictable failure before the first paid call (6cf8794)
compare saved evaluations against declared gates, with overlap passes and a completeness matrix (eb8ee5e)
compare: add paired task bootstrap bounds and sign tests (291d0ed)
compare: add paired task bootstrap bounds and sign tests (b211cbe)
judge: add blinded pairwise judging of two arms (fda27cc)
judge: add blinded pairwise judging of two arms (dac44d3)
judge: prepare a run that has no review packets from its saved snapshots (5c07532)
judge: prepare a run that has no review packets from its saved snapshots (8d1d3df)
keep attempt states in a SQLite ledger and resume interrupted studies (65bdb82)
keep comparison identity and caches when the output folder changes (02f5d3c)
make true-myth per-test coverage repeatable (0f4a7eb)
measure redundant added tests on csstree and true-myth with per-test coverage profiles (721a657)
measure redundant tests from LCOV with a per-test sidecar (dd7221d)
notify when a detached run starts coding, finishes, or fails (09b9d60)
notify when a detached run starts coding, finishes, or fails (86ad66f)
qualify a DeepSWE vanilla screening panel (942f12c)
qualify a DeepSWE vanilla screening panel (3df85d5)
record Mocha per-test coverage for csstree (31576ac)
record the run owner in run.lock and stop only its process group (ccb5e63)
record Vitest per-test coverage for true-myth (a7e4df3)
report overlap passes as their own correctness state (8722cb6)
resume a run directory with pluginbench run and run in the background (fe88473)
resume interrupted runs without repeating paid work, and stop them safely (1e9edbf)
share caches and comparison identity across output folders (57eb5b6)
show a task × measure × package completeness matrix in reports (ca2bbb1)
Bug Fixes
a run can resume after a kill during measurement or a stop during a settings write (3f1df1f)
accept a run path that goes through a parent directory or a symlink (91ec822)
accept a run path that goes through a parent directory or a symlink (3ac96e4)
accept coverage.py's module-entry line and Linux's extra Oxc binding (b4f8f5d)
accept the plugins built into the Claude CLI, and name the missing credential variables (643bdf8)
accept the plugins built into the Claude CLI, and name the missing credential variables (b5ecf55)
admit reviewed DeepSWE contracts that omit the collect service (8957b9a)
admit reviewed DeepSWE contracts that omit the collect service (c6a6909)
admit reviewed tasks under any umask and parse Fallow's version line (f169d76)
an overlap verdict with no existing test passing is unknown, not a fail (fd7ff9c)
an overlap verdict with no existing test passing is unknown, not a fail (353cb87)
check loads the saved measurement profiles the way a run does (47dcd2b)
check loads the saved measurement profiles the way a run does (cf37db3)
check measurement capacity for the containers that run at once (b4b0b3f)
classify a saved overlap fail again when compare or judge reads it (06a60bb)
classify withheld results in status the same way as the report (c97a069)
classify withheld results in status the same way as the report (565ec5f)
close the run ledger when a resume fails before coding, and name the cache when a finished repeat is not cached (f6e8d2b)
compare refuses to gate what is not comparable (454bbdc)
compare: refuse incomparable runs and leave doubtful gates undetermined (63ed7db)
count forked subagent usage from Codex 0.167.0 rollouts (e9f5c8a)
decide lock ownership from this process, not the owner record (4bdac81)
decide whether coding is left after reconciliation, and finish without the vanilla cache (da0b4af)