v0.7.0

v0.7.0 (2026-07-10)

This release is published under the MIT License.

Features

• Run-all multi-suite harness, validation, and reporting suite (a282bc9)

Comprehensive update folding together the run-all multi-suite layer, static validation, expanded reporting, and side-effect contract enforcement.

Run-all / multi-suite: - concurrent eval-suite execution with paired skill<->suite discovery - recursive eval discovery, changed-only mode, plugin eval layout - dry-run planning without agent calls, planned-case preview, plan totals - cost/duration totals, pass-rate gating, JUnit reports - failed-case rerun commands with context preservation - artifact archive + cross-run compare

Validation & assertions: - strict eval-suite linting and structured-field validation - negative artifact assertions, command-assertion false-positive fix - run-all assertion-failure reporting

Grading: - constrained OpenWebUI grader backend (localhost default)

Side effects & reporting: - enforced side-effect contracts, GitLab cleanup manifests - static HTML reports, skill portfolio report - skill-hash + run-trajectory recording - doctor command, JSON discovery output

New modules: fixtures.py, validation.py.

Corporate-specific fixtures removed; tool is fully generic. Default LLM grader backend is OpenRouter (public).

Co-Authored-By: Claude noreply@anthropic.com

---

Detailed Changes: v0.6.1...v0.7.0

v0.6.1

v0.6.1 (2026-06-12)

This release is published under the MIT License.

Bug Fixes

• Opencode non-cached input semantics + LLM grader retry on dropped assertions (f0939bc)

OpenCode reports input_tokens excluding cache reads (live runs show cached consistently larger than input), so the codex-style subtraction clamped non_cached_input_tokens to a bogus 0 and capped cached_pct at 100%. Override the split like claude-code.

The LLM judge intermittently drops an assertion from its JSON answer, which skipped the assertion and failed strict e2e runs. Retry once with only the dropped assertions before skipping; matched results are now also deduplicated and filtered to the requested assertions so a retry cannot double-count.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

• Tolerate malformed judge rows, correct cached_pct denominator (review feedback) (45b2677)

A malformed individual result row (non-dict, missing keys, non-string text) now only loses that row — its assertion goes to the retry instead of crashing the whole attempt and skipping everything.

cached_pct now divides by non_cached + cached, the only denominator valid for both reporting styles; cached / max(input, cached) showed a bogus 100% for claude-code/opencode.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

---

Detailed Changes: v0.6.0...v0.6.1

v0.6.0

v0.6.0 (2026-06-12)

This release is published under the MIT License.

Bug Fixes

• Harden subprocess decoding and null-safe artifact reads (review feedback) (4680b8e)

• errors="replace" on all subprocess output decoding (hooks, agent run,
  CLI version probe) so non-UTF-8 output can't crash the harness

• treat present-but-null JSON fields as defaults in summary token totals
  and the status command

• reject a directory passed as --pricing-config

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Documentation

• Add ROADMAP.md with planned harness improvements (4e50954)

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Features

• harness: Real-world feedback batch — targeted runs, budgets, hooks, observability (4641623)

Implements the confirmed work items from evaluating a live side-effect skill (commit/push/MR workflow) with this harness:

• --eval-id on run and validate: filter the suite before task creation, so all artifacts describe only the selected cases; unknown ids fail fast before any model call - per-case budget guards (--max-input/non-cached-input/output/total/ reasoning-tokens-per-case, --max-cost-per-case, --max-duration-per-case) with warn|fail|stop-suite actions, checked post-run - lifecycle hooks: --pre-run-command (aborts suite on failure before any model call), --post-grade-command (stdout JSON merges into grading.json), --post-run-command; all receive structured ASE_* env metadata - live progress lines plus progress.jsonl per-run event stream - summary.json per iteration plus status command; report --failures-only - cost semantics: cost_usd null means unavailable (not $0), cost_usd_source tracking, --pricing-config for CLIs that report no cost, OpenRouter reconciliation for claude-code, cached vs non-cached input split - reasoning token persistence (codex turn.completed, opencode step_finish); null
  means unreported, not zero - --reasoning-effort passed through to codex (model_reasoning_effort); unsupported agents warn and record null - run_meta.json records effective config: model, reasoning effort, base URL, agent CLI version, harness version

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

---

Detailed Changes: v0.5.0...v0.6.0

v0.5.0

v0.5.0 (2026-06-11)

This release is published under the MIT License.

Bug Fixes

• harness: Harden OpenRouter pricing fetch (review feedback) (d40feba)

Guard the pricing cache with a lock (eval threads run concurrently and could fire redundant fetches), send a real User-Agent (CDN-fronted APIs commonly 403 urllib's default), and type-check the response shape before indexing into it.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

• runner: Only record eval-created open PRs in cleanup manifest (d588d1c)

The PR snapshot is taken with gh pr list --state all, so _build_cleanup_entry could record PRs that were merged or closed during the run, and even PRs someone else opened on the source repo mid-run — cleanup would then try to close a stranger's PR with --delete-branch.

Record a new PR number only when the PR is still OPEN and its head branch is one this run pushed. Snapshots from before state/headRefName were captured keep the old behavior.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Chores

• examples: Expand sample_change.py into a realistic fixture (fdb3a4a)

The two-line print fixture gave agents nothing plausible to commit. The new fixture is a small user-directory module whose contents match what the eval prompts claim about it: a login fix (email normalization in authenticate) and a new search feature (search_users).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Documentation

• Real author metadata, python -m usage, cost reporting, agent comparison (03abf2a)

• pyproject authors: replace the placeholder with the real maintainer

• README: document running the CLI as python -m agent_skill_eval

• README: fix the stale codex invocation row (--full-auto was replaced
  by --sandbox workspace-write --skip-git-repo-check)

• README: document cost_usd semantics and the OpenRouter reconciliation
  (cost_usd_source / cost_usd_cli in timing.json)

• README: add a multi-agent comparison walkthrough showing how to read
  per-agent with/without-skill deltas from the report and benchmark.json

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

• todo: Mark TODO batch done — cost reconcile, tests, docs, cleanup (2143eb0)

Local-only cleanup also performed (not visible in the diff): stale merged local branches deleted, both .claude/worktrees copies of the old skill_eval module tree removed, stale untracked HANDOFF.md deleted.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Features

• harness: Reconcile claude-code cost against OpenRouter pricing (5246951)

The claude CLI prices runs at Anthropic list prices regardless of the endpoint it talks to, so cost_usd was wrong whenever the harness routed claude-code through OpenRouter. The generation API would give exact billed amounts but needs per-request ids the CLI does not expose; instead, recompute from the run's token counts and OpenRouter's published per-model rates (fetched once per process from the public models endpoint).

timing.json now records cost_usd_source ("cli", "openrouter-pricing", or "cli-unreconciled" when the model cannot be mapped to an OpenRouter slug) and keeps the CLI's original estimate in cost_usd_cli. Cache creation tokens are now captured too so the reconciled cost includes cache-write charges.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Testing

• harness: Pin claude-code, opencode, and codex CLI invocations (c224898)

build_command was only pinned for codex (and only the trust-check flag). Pin the full argv for all three real harnesses so a flag regression fails loudly, and cover cache_creation_input_tokens parsing.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

---

Detailed Changes: v0.4.1...v0.5.0

v0.4.1

v0.4.1 (2026-06-11)

This release is published under the MIT License.

Bug Fixes

• Todo batch — grader honesty, cost reporting, curated changelog (#11, 33ab02c)

• fix: grader honesty — skip-with-warning, workspace fallback warning, behavior-based negative control

• LLMGrader: grading errors and missing per-assertion results are now skipped (not failed) and warned about on stderr, so grader failures never masquerade as agent failures. - Workspace resolution for file-based checks is shared between the deterministic and LLM graders and warns (once per run dir) when it falls back to guessing from the output directory layout. - fix-failing-tests negative-control: reword the grader-run-dependent 'output contains the source code' assertion to the behavior that matters (file read and displayed/described).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

• feat: cost column in report, validate flags LLM-rubric assertions, curated changelog

• benchmark.json: per-config cost_usd mean/stddev and per-agent cost deltas; report gains a Cost (USD) column (table and markdown). - validate: classify_assertion() reports which assertions match no deterministic pattern and will be graded by the LLM rubric, instead of that happening silently at run time. - CHANGELOG.md: restore curated pre-0.4 entries (PSR-style headings); semantic-release now runs in insertion-only update mode with vendored templates that render commit summary lines only — no more commit bodies or Co-Authored-By trailers leaking into the changelog. - Version is single-sourced from pyproject.toml (PSR version_variables removed; init falls back to 0.0.0 when not installed). - TODO.md: tick everything done in this batch and items verified already done on main.

• fix: normalize assertion text when matching LLM grader responses

LLMs echo assertions with minor drift (casing, trailing periods, whitespace); exact-match made those falsely count as missing results. Addresses Gemini review on PR #11.

• chore: remove vendored PSR templates, use defaults

---

Co-authored-by: Claude Fable 5 noreply@anthropic.com

Documentation

• Update TODO — v0.4.0 state, subagent evals next, agent working notes (4b5b76f)

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

---

Detailed Changes: v0.4.0...v0.4.1

v0.4.0

v0.4.0 (2026-06-11)

This release is published under the MIT License.

Bug Fixes

• Address Gemini review findings (641b98b)

• cost accumulation test: float-safe compare via pytest.approx

• Makefile: default CLAUDE_CODE_MODEL to a real Anthropic ID so the
  target works even without config/baseline.env

• test-live: export model vars so e2e tests use configured models
  instead of hardcoded fallbacks

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Continuous Integration

• Replace tag-triggered publish with semantic-release on main (64a4cca)

Pushing to main now handles version bump, tag, GitHub release, and PyPI publish (trusted publishing) automatically from conventional commits — no manual tag push. Pattern borrowed from gcpath.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Features

• Codex harness fix, cost_usd metric, e2e milestone tests (bb34a28)

• CodexHarness: replace deprecated --full-auto with --sandbox workspace-write and add --skip-git-repo-check so codex runs in freshly git-inited eval workspaces without its trust prompt. - TimingData: new cost_usd field; claude-code parses total_cost_usd, opencode sums per-step cost. Cost, time, and accuracy are now all machine-readable per run for downstream skill optimization. - tests/test_e2e.py: two-tier end-to-end tests exercising the real CLI as a subprocess. Free tier (fake harness, no network) runs in every pytest invocation; live tier (claude-code, opencode, codex + LLM grader) is opt-in via the live marker and locks in the all-harnesses-complete-with-real-grades milestone. - Makefile: AGENTS variable, full-eval (includes codex), test-e2e and test-live targets; config/baseline.env pins claude-haiku-4-5-20251001 for claude-code (claude CLI validates --model client-side, so OpenRouter-only slugs are rejected).

Verified: make test-live 7/7 passed; 232 unit tests; lint clean.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

• Converge all names on agent-skill-eval, add ase alias (33f1084)

Repo (skill-evals), CLI (skill-eval), and module (skill_eval) now all match the PyPI package name agent-skill-eval. Short alias ase installs alongside the full command. Env var prefix SKILL_EVAL_* renamed to ASE_*. Version 0.4.0.

Skills remain the only eval target for now; subagent evals are noted as coming soon in the README.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

---

Detailed Changes: v0.3.0...v0.4.0