Skip to content

Releases: neozenith/pytest-xharness-eval

v0.2.0

Choose a tag to compare

@neozenith neozenith released this 23 Aug 13:03

Breaking

  • run.usage.total_tokens is now run.usage.accumulative_billed_tokens (ADR 0029): the four priced tiers summed over every model call. Update graders that assert on it.
  • .skillignore files are no longer read. Declare skill files that are not decision surface in the xharness_skill_ignore ini key; a bare pattern applies to every skill, <skill>: <pattern> to the skills the selector matches (ADR 0026).
  • Results and history.jsonl lines written by 0.1.x carry old keys until uv run -m pytest_xharness_eval.replay <skill>/evals/captured is run; replay rebuilds them from the captured logs without spending anything (ADR 0023).

New

  • A per-call ledger on every result, with context-window, TTFT and tokens-per-second metrics (ADRs 0019, 0024) and the case that produced the run (0025).
  • captured/report.html: a self-contained static microsite over the captured JSON, now built from the report-ui/ workspace (Vite, React, TypeScript, Tailwind, shadcn/ui, Recharts) with no CDN at view time; themed by design tokens; --xharness-report-inline embeds everything so it opens over file:// (ADRs 0020, 0021, 0028).
  • Session-log record kinds and skill file coverage per run (0022); coverage follows the shell's working directory across Bash calls (0027).
  • docs/token-accounting.md: how accumulative_billed_tokens and peak_context_tokens are derived, quoted from the providers' documentation and cross-checked against the harnesses' own totals.

Fixed

  • Codex cache_write_input_tokens is subtracted from OpenAI's inclusive input_tokens, so each tier prices once.
  • The report no longer shows the cross-turn billed sum beside a context percentage as if they were one quantity.

v0.1.1

Choose a tag to compare

@neozenith neozenith released this 21 Aug 05:59

Full Changelog: v0.1.0...v0.1.1

v0.1.0

Choose a tag to compare

@neozenith neozenith released this 21 Aug 05:48