TraceBench 0.4.0 — explicit recorded-observation budgets

TraceBench 0.4.0 adds explicit budgets over supplied recorded observations.

• Separate tracebench policy JSON/Markdown verdicts with exact pass-rate, p95-latency and total-cost thresholds.
• Missing/partial observations and currency mismatch remain explicit and reject selected rules.
• Optional reusable-workflow rules retain all eight reports before the combined gate. The final gate checks hashes and recomputes policy from its copied rules snapshot.
• Existing evaluation schema, reports and 0/1/2/3 behavior remain unchanged; policy acceptance cannot override an evaluation failure or regression.

Validation: 106 tests on Python 3.13/3.15 locally and hosted 3.11/3.13; fresh installed source/wheel/sdist; actual Chromium; independent decimal boundary review and normalization repair; downloaded five/eight-file hosted artifacts. A real normal consumer uploaded eight reports then failed with policy exit 1 while evaluation passed. Main CI and Pages passed at the release commit.

All fixture outputs, prices, token counts and latencies are synthetic/supplied. No model/provider is called or authenticated; these budgets do not establish live performance or model quality. Review and trust the pinned workflow/source and recorded inputs.

The attached wheel is the exact tested artifact. The source archive includes the final fixed implementation and matching immutable workflow/source pins. SHA256SUMS.txt identifies both assets. No PyPI publication is implied.

TraceBench 0.3.0 — Reusable recorded-evaluation CI

TraceBench 0.3.0 adds a reusable GitHub Actions workflow for recorded-output regression gates. It uploads JSON, Markdown, HTML and provenance before applying the strict gate, so a valid regression fails visibly while its reports remain available. Invalid inputs stay distinct from failed checks and regressions.

Use the pinned consumer workflow · Synthetic HTML report · Verification · PR #3

Inputs are validated workspace-relative files passed through environment values and argv. The workflow checks out immutable reviewed source with read-only permissions, stages a bounded explicit artifact set and revalidates its hashes/status before gating. Generated JSON reports use the documented 64 MiB bound in both stages; baseline input remains 16 MiB and workflow input files 10 MiB. The external example pins fixed source ancestor 9cfe62a4afd613fc9a15e24018cdf77bc8f65977, retained in main history.

Validation: 77 Python tests, independent review and report-size repair review; real installed-CLI 0/1/3 outcomes for reports between 16 and 64 MiB and upper/input-bound rejection; fresh source/wheel/sdist installs outside checkout. Hosted Python 3.11/3.13, clean/expected-regression and actual Chromium jobs passed. Downloaded report ZIPs matched service digests, exact file sets, gate hashes and source provenance. A separate normal consumer probe uploaded its two-regression report and then failed with exit 3. Final wheel runtime bytes and source-archive files match the integrated release source.

Python runtime remains standard-library only. Playwright/Node are pinned development tooling. Existing evaluation JSON/Markdown/HTML schemas, default exit meanings and synthetic public sample are preserved. All observations and prices are invented fixtures; no provider call, model-quality benchmark, baseline authenticity, production adoption or security audit is implied. The workflow executes trusted code and is not an arbitrary-code sandbox. No PyPI publication is implied.

TraceBench 0.2.0 — Standalone HTML evaluation reports

TraceBench 0.2.0 adds a standalone HTML report for recorded-output evaluations, with complete case/check results, exact numeric text, search, status filters, keyboard access and print support. HTML works without JavaScript; the optional controls use a fixed hash-restricted script. Recorded text is rendered as inert text with no external assets.

tracebench evaluate --cases fixtures/cases.jsonl --outputs fixtures/clean.jsonl --pricing fixtures/pricing.json --format html --report clean.html

Open the synthetic report · CLI and report contracts · Verification

Validation: 65 Python tests; independent code review and boundary fixes; hosted Python 3.11/3.13 CI; fresh wheel/source installs; real installed CLI outcomes 0/1/2/3; Chromium desktop/mobile, filters, keyboard, no-JavaScript, print and hostile-text checks. The final wheel is byte-identical to the installed verified candidate and its runtime modules match merged main. The public report matches the reproducible synthetic sample byte-for-byte. Main-only Pages publication was also tested against an overlapping non-main dispatch.

All outputs, observations and prices are synthetic. Standard-library Python runtime; no provider calls or model-quality benchmark. This is an educational offline evaluator, not a production security audit or adoption claim. HTML only reports the supplied evaluation; it does not authenticate baseline provenance. No PyPI publication is implied. Existing JSON/Markdown schemas and exit meanings are preserved.

TraceBench 0.1.0: offline evaluation and regression gates

Initial educational release of TraceBench: an offline evaluator for recorded AI outputs.

• Strict JSONL schemas and exact case-ID joins.
• Literal exact/contains/JSON-subset checks with English and Mandarin fixtures.
• Explicit Decimal token pricing and coverage-aware latency summaries.
• Compatible-suite baseline comparison with reproducible exit codes 0/1/2/3.
• JSON and inert Markdown reports; synthetic regression example included.
• Standard-library runtime and no-download wheel/source builds.

Verification: 49 automated tests; clean source and source-archive installs; installed console script exercised outside the checkout. GitHub Actions passes on Python 3.11 and 3.13. The independent review's two validation findings have regression tests and fixes. See docs/VERIFICATION.md.

All fixtures, prices and observations are synthetic. No provider calls, model-quality benchmark, production adoption or security audit is implied. Baseline authenticity and editable installation are outside this release's scope.

Install a downloaded wheel with python -m pip install tracebench-0.1.0-py3-none-any.whl, then run tracebench --help. Clone the repository for fixtures and the complete example walkthrough.