Repository navigation
v0.17.0 — eval baseline arm
The benchmark gains its missing comparison: a bare-agent baseline arm, so "context economy pays" is a measured claim, not an argument.
Added
evals/run-eval.sh --baseline— bare-agent arm: no plugin loads, the workspace copy is stripped of the fixture's methodology onboarding (strip verified fail-closed), driven by a new frozen prompt (evals/feature-prompt-baseline.md) whose feature paragraph is byte-identical to the LCD prompt's (test-enforced), with anEVAL-DONEfinish line.--baseline+--plugin-diris refused as a contradiction.evals/grade.sh --baseline— grades bare workspaces with the identical row shape (audit: n/a); fail-closed both ways. Golden-locked grading criteria unchanged.tests/test-eval-baseline.sh— fixture tests for all of the above.
Changed
- Both eval arms now run config-isolated, symmetrically:
CLAUDE_CONFIG_DIRat a fresh empty dir +--settings '{"enabledPlugins":[]}', so user-level plugins and the personal~/.claude/CLAUDE.mdno longer leak into benchmark runs. The arms differ by exactly one flag:--plugin-dir. Rows recorded before this isolation are not comparable with rows after it.
Full changelog: CHANGELOG.md