Releases: neozenith/pytest-xharness-eval
Releases · neozenith/pytest-xharness-eval
Release list
v0.2.0
Breaking
run.usage.total_tokensis nowrun.usage.accumulative_billed_tokens(ADR 0029): the four priced tiers summed over every model call. Update graders that assert on it..skillignorefiles are no longer read. Declare skill files that are not decision surface in thexharness_skill_ignoreini key; a bare pattern applies to every skill,<skill>: <pattern>to the skills the selector matches (ADR 0026).- Results and
history.jsonllines written by 0.1.x carry old keys untiluv run -m pytest_xharness_eval.replay <skill>/evals/capturedis run; replay rebuilds them from the captured logs without spending anything (ADR 0023).
New
- A per-call ledger on every result, with context-window, TTFT and tokens-per-second metrics (ADRs 0019, 0024) and the case that produced the run (0025).
captured/report.html: a self-contained static microsite over the captured JSON, now built from thereport-ui/workspace (Vite, React, TypeScript, Tailwind, shadcn/ui, Recharts) with no CDN at view time; themed by design tokens;--xharness-report-inlineembeds everything so it opens overfile://(ADRs 0020, 0021, 0028).- Session-log record kinds and skill file coverage per run (0022); coverage follows the shell's working directory across Bash calls (0027).
docs/token-accounting.md: howaccumulative_billed_tokensandpeak_context_tokensare derived, quoted from the providers' documentation and cross-checked against the harnesses' own totals.
Fixed
- Codex
cache_write_input_tokensis subtracted from OpenAI's inclusiveinput_tokens, so each tier prices once. - The report no longer shows the cross-turn billed sum beside a context percentage as if they were one quantity.
v0.1.1
v0.1.0
Full Changelog: https://github.com/neozenith/pytest-xharness-eval/commits/v0.1.0