Skip to content

v0.5.0

Choose a tag to compare

@github-actions github-actions released this 11 Nov 17:41
b2bf866

0.5.0 (2026-11-11)

⚠ BREAKING CHANGES

  • leave a mean gate undetermined when an attempt has no value
  • skip a measure that is broken on some tasks, and stop when a measure is ready on no task
  • run takes the config or run directory only as a positional argument. --config, --resume, --jobs, --guidance-name and every evaluation flag (--harness, --model, --reasoning, --guidance, --skill, --plugin, --tasks, --task, --measure, --review-*, --repeats, --seed, --timeout) are removed; set them in the config. run --dry-run prints a summary instead of JSON unless --json is given. The plan's baseline key is now vanilla. examples/evaluation.yaml moved to src/pluginbench/evaluation.example.yaml.

Features

  • accept per-test numbits sidecars for LCOV (87d8b64)
  • add pluginbench compare to check gates across saved evaluations (c7031f9)
  • add status, review and init; take the config only as a positional argument (eee56a0)
  • cache each finished vanilla attempt as soon as it completes (11c368a)
  • check every predictable failure before the first paid call (e3ff981)
  • check every predictable failure before the first paid call (6cf8794)
  • compare saved evaluations against declared gates, with overlap passes and a completeness matrix (eb8ee5e)
  • compare: add paired task bootstrap bounds and sign tests (291d0ed)
  • compare: add paired task bootstrap bounds and sign tests (b211cbe)
  • judge: add blinded pairwise judging of two arms (fda27cc)
  • judge: add blinded pairwise judging of two arms (dac44d3)
  • judge: prepare a run that has no review packets from its saved snapshots (5c07532)
  • judge: prepare a run that has no review packets from its saved snapshots (8d1d3df)
  • keep attempt states in a SQLite ledger and resume interrupted studies (65bdb82)
  • keep comparison identity and caches when the output folder changes (02f5d3c)
  • make true-myth per-test coverage repeatable (0f4a7eb)
  • measure redundant added tests on csstree and true-myth with per-test coverage profiles (721a657)
  • measure redundant tests from LCOV with a per-test sidecar (dd7221d)
  • notify when a detached run starts coding, finishes, or fails (09b9d60)
  • notify when a detached run starts coding, finishes, or fails (86ad66f)
  • qualify a DeepSWE vanilla screening panel (942f12c)
  • qualify a DeepSWE vanilla screening panel (3df85d5)
  • record Mocha per-test coverage for csstree (31576ac)
  • record the run owner in run.lock and stop only its process group (ccb5e63)
  • record Vitest per-test coverage for true-myth (a7e4df3)
  • report overlap passes as their own correctness state (8722cb6)
  • resume a run directory with pluginbench run and run in the background (fe88473)
  • resume interrupted runs without repeating paid work, and stop them safely (1e9edbf)
  • share caches and comparison identity across output folders (57eb5b6)
  • show a task × measure × package completeness matrix in reports (ca2bbb1)

Bug Fixes

  • a run can resume after a kill during measurement or a stop during a settings write (3f1df1f)
  • accept a run path that goes through a parent directory or a symlink (91ec822)
  • accept a run path that goes through a parent directory or a symlink (3ac96e4)
  • accept coverage.py's module-entry line and Linux's extra Oxc binding (b4f8f5d)
  • accept the plugins built into the Claude CLI, and name the missing credential variables (643bdf8)
  • accept the plugins built into the Claude CLI, and name the missing credential variables (b5ecf55)
  • admit reviewed DeepSWE contracts that omit the collect service (8957b9a)
  • admit reviewed DeepSWE contracts that omit the collect service (c6a6909)
  • admit reviewed tasks under any umask and parse Fallow's version line (f169d76)
  • an overlap verdict with no existing test passing is unknown, not a fail (fd7ff9c)
  • an overlap verdict with no existing test passing is unknown, not a fail (353cb87)
  • check loads the saved measurement profiles the way a run does (47dcd2b)
  • check loads the saved measurement profiles the way a run does (cf37db3)
  • check measurement capacity for the containers that run at once (b4b0b3f)
  • classify a saved overlap fail again when compare or judge reads it (06a60bb)
  • classify withheld results in status the same way as the report (c97a069)
  • classify withheld results in status the same way as the report (565ec5f)
  • close the run ledger when a resume fails before coding, and name the cache when a finished repeat is not cached (f6e8d2b)
  • compare refuses to gate what is not comparable (454bbdc)
  • compare: refuse incomparable runs and leave doubtful gates undetermined (63ed7db)
  • count forked subagent usage from Codex 0.167.0 rollouts (e9f5c8a)
  • decide lock ownership from this process, not the owner record (4bdac81)
  • decide whether coding is left after reconciliation, and finish without the vanilla cache (da0b4af)
  • default jobs to 2 and cap it at 4 (f46b052)
  • deny writes outside the granted roots in the Linux sandbox (7fef526)
  • deny writes outside the granted roots in the Linux sandbox (b05b459)
  • finish a run interrupted after its last attempt, and test stop in its own session (5368489)
  • grade an attempt whose workspace holds a symlink the agent left behind (febdf03)
  • grade an attempt whose workspace holds a symlink the agent left behind (a70160d)
  • judge: refuse a tally when a requested judge never started (4841a09)
  • keep a background run alive when its launcher's cgroup ends (f566e3a)
  • keep a fail the receipt recorded unless the reward was 0 (710fde9)
  • keep a run going when a Harbor worker overruns its budget (d5edaaf)
  • keep a run going when a Harbor worker overruns its budget (038d472)
  • keep a task's own git parameters, and keep git's error when source qualification fails (2259438)
  • keep cache lookups working when saved entries are partial, malformed or pruned (9632e77)
  • keep reasons for unmeasured tasks and print every missing-value note (66fa85d)
  • keep same-title JS test identities stable across snapshots (8e365de)
  • keep saved runs and caches usable when Docker drifts or PluginBench is updated (d5c81ac)
  • keep saved runs usable when Docker memory moves or PluginBench is updated (29c5710)
  • keep the during-collection observer check for profiles from an earlier version (9474664)
  • leave a mean gate undetermined when an attempt has no value (ab3a096)
  • leave the decision to resume with run, and name only the command (6e199cf)
  • let isolated Codex reviews start the code-mode host (40ed0b8)
  • lift Mocha's hook timeout in the csstree per-test coverage run (940c8cf)
  • lift Mocha's hook timeout in the csstree per-test coverage run (8c103a9)
  • load saved coverage profiles whose inline script is longer than a file name (fd0de97)
  • load saved coverage profiles whose inline script is longer than a file name (a158c19)
  • make earlier builds refuse a run that needs the supplied git identity (83c793c)
  • make status count receipts without totals and never disturb or misread a run (5556c66)
  • measure a task that has no collector instead of refusing the run (54f2986)
  • measure line changes when an unchanged file is not UTF-8 (a4dffbe)
  • measure line changes when an unchanged file is not UTF-8 (39ba91e)
  • merge same-title JS tests into one per-test identity (c2e4675)
  • name the measurement profile and the next step when an observer refuses it (6d82d5f)
  • name the whole missing treatment path, so an unfilled starter config shows its placeholder (fa5c0fa)
  • offer a resume only where run accepts it, and quote the command (61675c6)
  • qualify oracle and nop through the paid export and patch path (dfd64d0)
  • read Rust dep-info that C build scripts continue across lines (f37ff0e)
  • read Rust dep-info that C build scripts continue across lines (959d5f6)
  • record a stopped run as stopped, not failed, and lock review before planning (6bd88d9)
  • refuse a resume before coding when measurement or review would fail (2dd3add)
  • refuse a run given twice to compare and list partly measured tasks with their reasons (6f70154)
  • refuse resumes whose task runtime changed, and finish runs with nothing left to code (c5aad14)
  • register a held lock inside the block that releases it (2ba1855)
  • report a run that stopped during coding, and say that the report is partial (9ea60af)
  • report a run that stopped during coding, and say that the report is partial (2b31809)
  • resume a run whose measurement was killed (76ef39e)
  • resume holds run.lock, and every held stop is delivered (836ec5f)
  • reuse prepared images and vanilla attempts after Docker or PluginBench changes (e1b1e8c)
  • save evaluation settings and their binding as one unit (25a3819)
  • say why measurement starts again after a kill (bbbea86)
  • say why the dead-code graph is incomplete (0fd95ee)
  • say why the dead-code graph is incomplete (e5d22cb)
  • show a killed run as interrupted and say how to resume it (7ae7634)
  • show a killed run as interrupted and say how to resume it (e62c7aa)
  • show why source qualification failed instead of "docker start failed" (28b6b1e)
  • show why the commit probe failed instead of "docker start failed" (5e190a3)
  • show why the commit probe failed instead of "docker start failed" (44a0f37)
  • skip a measure that is broken on some tasks, and stop when a measure is ready on no task (b201e41)
  • skip a saved cache entry whose lock file cannot be opened (7f2da0e)
  • skip unreadable or malformed saved cache entries, and count each measurement input's largest container (0bcaa92)
  • stop before writing preflight results when the run has moved (f68b3e2)
  • stop containers a killed attempt left running when the run resumes (ca084fa)
  • stop on Codex backend failures and keep background runs past their launcher (01597a8)
  • stop the containers a killed attempt left running when the run resumes (97058dd)
  • stop the run when a Codex session fails before its first model turn (09eaa0f)
  • supply a neutral git identity to tasks whose image has none (008a33d)
  • supply a neutral git identity to tasks whose image has none (a87c5d2)
  • survive kills during resume and never write to a moved run (093f526)
  • unblock Linux evaluations: modes, Fallow version, Oxc binding, coverage line 0 (9c7ae3f)
  • write no per-test sidecar after a failed sampling run (944a088)

Documentation

  • align the docs site with the sibling tool pages (af77f26)
  • build Astro documentation site in the shared tool frame (1a8227d)
  • correct the csstree same-title group count (4a54f5a)
  • describe background runs, stop, and resume (8926e8e)
  • generate references and document reproducible evaluations (cb044fe)
  • generate the config reference and metrics glossary, and add a reproduce guide (e3c7b1a)
  • judge: say tally checks every requested judge (7566a69)
  • link the config reference, metrics, and reproduce guide from the README (ddc9d82)
  • note AST remapping populations and readiness limit (cbff228)
  • publish the docs site on the shared tool-site stack (de2d23f)
  • rewrite the README around install, first run, and limits (7ce0419)
  • rewrite the README around install, first run, and limits (c1b65a9)
  • say that resume stops containers a killed attempt left running (a0abac1)
  • say the default /tmp is read-only in the Linux sandbox (097dbf0)
  • show how to test instructions, skills, and plugins as treatments (7714455)
  • show how to test instructions, skills, and plugins as treatments (797f306)
  • site: render Markdown samples with the library code block (64ee8dd)
  • site: rewrite links inside tables too (389b4bd)
  • site: wrap install commands, number repeated table names, detect shell continuations exactly (a8354e1)
  • state cache precedence, the earlier-build guarantee and the resume checks as implemented (119f583)
  • state how stop treats jobs 1, when broken measures block, and when the host sandbox is needed (9353654)
  • state the guidance size limit and what skill and plugin packages may not contain (7539b54)
  • state the limits of best-effort caching and of the earlier-build refusal (321c615)