v0.8.0

0.8.0 (2026-11-13)

Features

• accept paths on Claude plugin skills (#93) (7641071)
• add measures to a finished run with pluginbench measure (#86) (b9b9d08)
• add pluginbench dashboard, a local read-only web view of runs (#89) (03a25be)
• admit Claude plugin agents and prove their sources in discovery (#92) (acc8b36)
• force Claude subagents onto the runner model (#91) (195ee21)
• let Claude Code vanilla attempts use the verified vanilla cache (#84) (511ace2)
• let status observers skip the Docker container check (#88) (786abf9)
• list every attempt of every reported run in the dashboard (#90) (e9dc66f)

Bug Fixes

• accept a relative tasks.source that climbs out of the config directory (#80) (89bc629)
• authenticate Claude reviews and judges with the subscription token only (#82) (537bcd9)
• bound untranscribed Claude usage by each compaction's token counts (#83) (fa5ce5d)
• check hooked Codex usage against the kept session usage (#81) (e13ca57)
• let measurement of saved runs ignore Harbor and coding-transport changes (#85) (99e7205)
• refuse non-boolean Claude skill invocation flags at admission (#79) (393422d)

v0.7.1

0.7.1 (2026-11-13)

Bug Fixes

• admit Claude skills that only the user can invoke (#77) (23ab9d9)

v0.7.0

0.7.0 (2026-11-13)

Features

• admit any Claude CLI version for coding with a warning (#65) (0faba74)
• admit Claude plugins with Claude component rules (#67) (03c9d0d)
• deliver Claude treatments and prove discovery without a model call (#68) (3dfeb06)
• judge: bind verdicts to the judge CLI version (#70) (3bb6f70)
• launch Claude coding attempts through the pinned contract (#72) (f3478ad)
• validate saved Claude coding token usage on reuse and report (#71) (315540a)

Bug Fixes

• authenticate Claude coding with the subscription token, never an API key (#69) (270e180)
• clearer compare notes, vanilla in progress lines, explained judge agreement (#74) (2c1de6a)
• judge: parse judge evidence with the judge's own harness (#64) (1501d64)
• name no harness in the unvalidated usage note of status (#76) (53ab15f)
• validate reused raw artifacts without parsing JSON (#73) (017bd36)

Documentation

• document Claude Code as a coding harness (#75) (567dd2d)

v0.6.1

0.6.1 (2026-11-12)

Bug Fixes

• let status use the rebuilt report's token validation (#62) (3cc30e4)

v0.6.0

0.6.0 (2026-11-12)

Features

• accept new Codex CLI versions and validate token usage per attempt (#54) (0160f18)
• judge: bind verdicts and report bias and token usage (#56) (78324ef)
• make the coding CLI version a run input (#57) (8591493)

Bug Fixes

• drain active attempts when running a single job (#53) (d653259)
• explain stops and unconfirmable runs instead of going silent (#58) (0994314)
• name the vanilla arm vanilla in report outputs (#60) (0541137)
• revalidate saved token usage on reuse and report (#59) (a22afa7)
• run judges and reviews with the Codex standalone install, and refuse unable verdicts (#55) (ea5a5b6)

v0.5.1

0.5.1 (2026-11-11)

Documentation

• install from PyPI with pipx or uv (ce546f8)
• install from PyPI with pipx or uv (882b6f5)
• site: use the library's site header and YAML highlighting from @filipgutica/ui 1.34.0 (413da50)
• site: use the shared site header from @filipgutica/ui 1.34.0 (e39864d)

v0.5.0

0.5.0 (2026-11-11)

⚠ BREAKING CHANGES

• leave a mean gate undetermined when an attempt has no value
• skip a measure that is broken on some tasks, and stop when a measure is ready on no task
• run takes the config or run directory only as a positional argument. --config, --resume, --jobs, --guidance-name and every evaluation flag (--harness, --model, --reasoning, --guidance, --skill, --plugin, --tasks, --task, --measure, --review-*, --repeats, --seed, --timeout) are removed; set them in the config. run --dry-run prints a summary instead of JSON unless --json is given. The plan's baseline key is now vanilla. examples/evaluation.yaml moved to src/pluginbench/evaluation.example.yaml.

Features

• accept per-test numbits sidecars for LCOV (87d8b64)
• add pluginbench compare to check gates across saved evaluations (c7031f9)
• add status, review and init; take the config only as a positional argument (eee56a0)
• cache each finished vanilla attempt as soon as it completes (11c368a)
• check every predictable failure before the first paid call (e3ff981)
• check every predictable failure before the first paid call (6cf8794)
• compare saved evaluations against declared gates, with overlap passes and a completeness matrix (eb8ee5e)
• compare: add paired task bootstrap bounds and sign tests (291d0ed)
• compare: add paired task bootstrap bounds and sign tests (b211cbe)
• judge: add blinded pairwise judging of two arms (fda27cc)
• judge: add blinded pairwise judging of two arms (dac44d3)
• judge: prepare a run that has no review packets from its saved snapshots (5c07532)
• judge: prepare a run that has no review packets from its saved snapshots (8d1d3df)
• keep attempt states in a SQLite ledger and resume interrupted studies (65bdb82)
• keep comparison identity and caches when the output folder changes (02f5d3c)
• make true-myth per-test coverage repeatable (0f4a7eb)
• measure redundant added tests on csstree and true-myth with per-test coverage profiles (721a657)
• measure redundant tests from LCOV with a per-test sidecar (dd7221d)
• notify when a detached run starts coding, finishes, or fails (09b9d60)
• notify when a detached run starts coding, finishes, or fails (86ad66f)
• qualify a DeepSWE vanilla screening panel (942f12c)
• qualify a DeepSWE vanilla screening panel (3df85d5)
• record Mocha per-test coverage for csstree (31576ac)
• record the run owner in run.lock and stop only its process group (ccb5e63)
• record Vitest per-test coverage for true-myth (a7e4df3)
• report overlap passes as their own correctness state (8722cb6)
• resume a run directory with pluginbench run and run in the background (fe88473)
• resume interrupted runs without repeating paid work, and stop them safely (1e9edbf)
• share caches and comparison identity across output folders (57eb5b6)
• show a task × measure × package completeness matrix in reports (ca2bbb1)

Bug Fixes

• a run can resume after a kill during measurement or a stop during a settings write (3f1df1f)
• accept a run path that goes through a parent directory or a symlink (91ec822)
• accept a run path that goes through a parent directory or a symlink (3ac96e4)
• accept coverage.py's module-entry line and Linux's extra Oxc binding (b4f8f5d)
• accept the plugins built into the Claude CLI, and name the missing credential variables (643bdf8)
• accept the plugins built into the Claude CLI, and name the missing credential variables (b5ecf55)
• admit reviewed DeepSWE contracts that omit the collect service (8957b9a)
• admit reviewed DeepSWE contracts that omit the collect service (c6a6909)
• admit reviewed tasks under any umask and parse Fallow's version line (f169d76)
• an overlap verdict with no existing test passing is unknown, not a fail (fd7ff9c)
• an overlap verdict with no existing test passing is unknown, not a fail (353cb87)
• check loads the saved measurement profiles the way a run does (47dcd2b)
• check loads the saved measurement profiles the way a run does (cf37db3)
• check measurement capacity for the containers that run at once (b4b0b3f)
• classify a saved overlap fail again when compare or judge reads it (06a60bb)
• classify withheld results in status the same way as the report (c97a069)
• classify withheld results in status the same way as the report (565ec5f)
• close the run ledger when a resume fails before coding, and name the cache when a finished repeat is not cached (f6e8d2b)
• compare refuses to gate what is not comparable (454bbdc)
• compare: refuse incomparable runs and leave doubtful gates undetermined (63ed7db)
• count forked subagent usage from Codex 0.167.0 rollouts (e9f5c8a)
• decide lock ownership from this process, not the owner record (4bdac81)
• decide whether coding is left after reconciliation, and finish without the vanilla cache (da0b4af)
• default jobs to 2 and cap it at 4 ([f46b052](https://github.com/fi...

PluginBench v0.4.0

PluginBench now runs every coding evaluation as a native Harbor task. Codex receives instructions, complete skills, and plugins through its native interfaces, and every treatment gets the same task prompt.

• Freeze treatment contents, file modes, runtime settings, and native discovery receipts. Isolate account plugins, and track root and child session usage without treating missing usage as zero.
• Grade, measure, review, and reuse vanilla results from one reconstructed canonical submission, with readiness and cleanup evidence.
• Report the code each submission adds: net cognitive and cyclomatic complexity, functions, files, hotspots, changed-line coverage, redundant added tests, and duplicated test lines. Measure each task's solution.patch offline as a reference baseline and report ratios to it.
• Rework the report with fixed arm order, grouped bars, a key-results table, passing-only summaries, and optional dated prices.json cost rows.
• Remove the local and Pier runners, bundled benchmark tasks, the multi-stage review flow, whole-snapshot complexity maxima, and the leave-one-out test-overlap measure. Report regeneration no longer reads multi-stage standards or ledger archives, and saved settings that select test overlap fail validation.

Install the attached Python wheel or source archive; see the README for Docker and Codex prerequisites and examples/evaluation.yaml. Run pluginbench run --config evaluation.yaml --dry-run before a paid evaluation. This remains experimental: the reference baseline and per-test redundancy have not yet run on a full saved evaluation.

PluginBench v0.3.0

PluginBench now compares text skills and instructions with a fresh vanilla baseline through two commands: run and report.

• Choose Codex or Claude, exact model and reasoning, named guidance packages, tasks, measurements, and a fixed reviewer in YAML, JSON, or flags.
• Run five supported DeepSWE tasks with bounded concurrency; export package-level baseline deltas and individual-task results as HTML, JSON, and CSV.
• Preserve saved evidence, distinguish missing measurements from zero, and reject invalid review citations or redirected report outputs.
• Remove the old Promptfoo/SWE-bench interface and obsolete commands. Align Node to 22.20.0 and exclude local benchmark caches from source distributions.

Install the attached Python wheel or source archive; see the README for task prerequisites and an example configuration. This remains experimental: text is supplied directly, plugin hooks and automatic skill activation are outside the evaluation, and paid Claude evaluations have not been validated.