Skip to content

PluginBench v0.3.0

Choose a tag to compare

View all tags
@filipgutica filipgutica released this 11 Oct 06:47
· 265 commits to main since this release
v0.3.0
65d751a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

PluginBench now compares text skills and instructions with a fresh vanilla baseline through two commands: run and report.

  • Choose Codex or Claude, exact model and reasoning, named guidance packages, tasks, measurements, and a fixed reviewer in YAML, JSON, or flags.
  • Run five supported DeepSWE tasks with bounded concurrency; export package-level baseline deltas and individual-task results as HTML, JSON, and CSV.
  • Preserve saved evidence, distinguish missing measurements from zero, and reject invalid review citations or redirected report outputs.
  • Remove the old Promptfoo/SWE-bench interface and obsolete commands. Align Node to 22.20.0 and exclude local benchmark caches from source distributions.

Install the attached Python wheel or source archive; see the README for task prerequisites and an example configuration. This remains experimental: text is supplied directly, plugin hooks and automatic skill activation are outside the evaluation, and paid Claude evaluations have not been validated.

Assets 5
Loading