Skip to content

v0.17.0 — eval baseline arm

Choose a tag to compare

@stid stid released this 23 Aug 00:37
· 53 commits to main since this release
ca41b35

The benchmark gains its missing comparison: a bare-agent baseline arm, so "context economy pays" is a measured claim, not an argument.

Added

  • evals/run-eval.sh --baseline — bare-agent arm: no plugin loads, the workspace copy is stripped of the fixture's methodology onboarding (strip verified fail-closed), driven by a new frozen prompt (evals/feature-prompt-baseline.md) whose feature paragraph is byte-identical to the LCD prompt's (test-enforced), with an EVAL-DONE finish line. --baseline + --plugin-dir is refused as a contradiction.
  • evals/grade.sh --baseline — grades bare workspaces with the identical row shape (audit: n/a); fail-closed both ways. Golden-locked grading criteria unchanged.
  • tests/test-eval-baseline.sh — fixture tests for all of the above.

Changed

  • Both eval arms now run config-isolated, symmetrically: CLAUDE_CONFIG_DIR at a fresh empty dir + --settings '{"enabledPlugins":[]}', so user-level plugins and the personal ~/.claude/CLAUDE.md no longer leak into benchmark runs. The arms differ by exactly one flag: --plugin-dir. Rows recorded before this isolation are not comparable with rows after it.

Full changelog: CHANGELOG.md