Skip to content

v0.4.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 11 Jun 11:48
· 24 commits to main since this release
v0.4.0
bb5808d

v0.4.0 (2026-06-11)

This release is published under the MIT License.

Bug Fixes

  • Address Gemini review findings (641b98b)

  • cost accumulation test: float-safe compare via pytest.approx

  • Makefile: default CLAUDE_CODE_MODEL to a real Anthropic ID so the
    target works even without config/baseline.env

  • test-live: export model vars so e2e tests use configured models
    instead of hardcoded fallbacks

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Continuous Integration

  • Replace tag-triggered publish with semantic-release on main (64a4cca)

Pushing to main now handles version bump, tag, GitHub release, and PyPI publish (trusted publishing) automatically from conventional commits — no manual tag push. Pattern borrowed from gcpath.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Features

  • Codex harness fix, cost_usd metric, e2e milestone tests (bb34a28)

  • CodexHarness: replace deprecated --full-auto with --sandbox workspace-write and add --skip-git-repo-check so codex runs in freshly git-inited eval workspaces without its trust prompt. - TimingData: new cost_usd field; claude-code parses total_cost_usd, opencode sums per-step cost. Cost, time, and accuracy are now all machine-readable per run for downstream skill optimization. - tests/test_e2e.py: two-tier end-to-end tests exercising the real CLI as a subprocess. Free tier (fake harness, no network) runs in every pytest invocation; live tier (claude-code, opencode, codex + LLM grader) is opt-in via the live marker and locks in the all-harnesses-complete-with-real-grades milestone. - Makefile: AGENTS variable, full-eval (includes codex), test-e2e and test-live targets; config/baseline.env pins claude-haiku-4-5-20251001 for claude-code (claude CLI validates --model client-side, so OpenRouter-only slugs are rejected).

Verified: make test-live 7/7 passed; 232 unit tests; lint clean.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

  • Converge all names on agent-skill-eval, add ase alias (33f1084)

Repo (skill-evals), CLI (skill-eval), and module (skill_eval) now all match the PyPI package name agent-skill-eval. Short alias ase installs alongside the full command. Env var prefix SKILL_EVAL_* renamed to ASE_*. Version 0.4.0.

Skills remain the only eval target for now; subagent evals are noted as coming soon in the README.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Detailed Changes: v0.3.0...v0.4.0

Assets 4
Loading