v0.5.0

TraceLens 0.5.0 makes evaluation results comparable and explainable. Every run records provenance (task content hashes, grader and adapter identity, runner settings) so two runs are checked for compatibility before they are compared; tracelens compare gives a verdict between two saved runs with a paired task bootstrap; tracelens inspect explains failed trials from a trials file; tracelens run --config tracelens.yaml replaces long flag lists; every command shares one exit-code contract; and the pass-rate, pass@k, and pass^k estimators were tightened so harness failures leave the denominator and unevaluable gates no longer pass. Releases are now prepared and published by the release pipeline.

Added

• One-click release preparation. The "Release prepare" workflow takes a
  version, validates it against the tags and the changelog, moves the
  [Unreleased] entries into a dated section (scripts/prepare_release.py,
  which refuses to release nothing), and opens a release: vX.Y.Z pull
  request with the rendered notes; merging it makes the "Release tag"
  workflow tag the merge commit and run the release workflow with
  publish=true. Nothing releases on an ordinary merge, and the manual
  tag path still works. (#88)
• tracelens report --format ci. Re-renders the one-line CI summary
  tracelens run printed, gate line included, from a saved results file,
  so a job summary or script can read it without parsing Markdown. report
  never re-decides the gate: it exits 0. (#75)
• Releases create their GitHub Release automatically. The release
  workflow now runs three jobs: build and verify (tag matches the built
  version; release notes rendered from the changelog's dated section by
  scripts/release_notes.py, failing before anything is published when the
  section is missing or empty), publish to PyPI (skip-existing so a
  re-run of an already-published tag is safe), and create the GitHub
  Release with those notes and the wheel and sdist attached, marked as a
  pre-release for any non-final version. The release step updates an
  existing release instead of failing, so re-runs after a partial failure
  are safe, and workflow_dispatch is a dry run that publishes nothing.
  docs/releasing.md gains the verification checklist and failure/re-run
  guidance. (#54)
• Positioning and contributor guidance aligned with what is actually
  demonstrated. The README and docs home lead with what TraceLens is
  (repo-owned local regression checks, inspectable artifacts, no backend,
  explicit uncertainty) and carry a "What is demonstrated today" table that
  separates the tested mechanism from the not-yet-published downstream
  evidence; the adjacent-tools page acknowledges that hosted platforms also
  run evaluations, experiments, and CI checks and names no vendor features;
  fingerprint explanations no longer claim exact reproducibility or causal
  attribution (DecisionSpec docstrings included); CONTRIBUTING.md
  describes the current extras ([datasets], [docs], PyYAML in core),
  the make verify gate, where things live, and three small first issues
  (#74, #75, #76). (#53)
• The documented user journey runs in CI from a built wheel.
  tests/journey/test_user_journey.py drives real tracelens processes
  through the documented workflow in a scratch project (an existing
  project, init, run --config, baselines from the README snippet, the
  gate enabled in tracelens.yaml, an intentional regression that blocks,
  inspect, compare, a --task-id rerun, an infra outage and a grader
  crash made unevaluable and told apart, malformed input and a bad config,
  checkpoint/resume re-executing nothing, report, sample), checking
  exit codes and persisted decisions at every step. A new CI job installs
  a freshly built wheel into a clean environment and runs the journey
  against its console script (TRACELENS_CLI). (#33, Stage A)
• tracelens inspect: explain failed trials from a trials file.
  tracelens inspect eval/results/trials.json --failures prints, per
  failing trial, one kind (agent failure, infra error, or grader crash,
  never conflated, harness causes first), status and attempts, expected
  versus actual output (--eval-set joins the task's name, input, and
  declared expectation), every grader's verdict, score, metrics, and
  feedback, and the transcript's steps, tokens, tool calls, and errors.
  Absent fields read missing; output is bounded (400 characters per
  field, 20 steps per transcript) with an explicit count of what was
  omitted, and --full lifts the bounds. Filters: --kind agent|infra| grader|not-run|passed, --task-id, --grader (trials that grader
  failed or crashed on), --all, --limit. --html writes a
  self-contained, escaped, offline drilldown that reads on a phone;
  --json writes the same view as data. The command exits 0 whenever the
  file was read (it reports, the gate decides) and 2 on input errors.
  tracelens run --task-id ID ... (run.task_ids in tracelens.yaml)
  reruns only the named tasks and refuses unknown ids; its provenance and
  checkpoint identity cover the subset. New guide: "Debugging a Failed
  Evaluation". (#52)
• tracelens compare: a verdict between two saved runs.
  tracelens compare baseline-trials.json candidate-trials.json (and
  compare_runs() in Python) implements the statistical contract's
  run-versus-run section: tasks are aligned by content through the runs'
  provenance (changed, added, or removed tasks and different graders make
  the runs incompatible; --unmatched-tasks exclude compares the shared
  tasks and lists the rest; artifacts without provenance align by id and
  are labelled, or refused with --require-provenance), one statistic per
  task and run is paired (--metric pass_rate | mean_score | <grader_id>.<metric_name>, --direction lower for latency-like metrics,
  --grader for multi-grader runs), and the mean paired difference gets a
  percentile bootstrap over tasks, a sign-flip p-value (exact for small
  suites), and a verdict against --threshold: improvement, equivalent
  within the threshold, or significant but below it exit 0; regression
  exits 1; inconclusive or insufficient evidence exits 2 (--observe
  forces 0). The terminal summary ("what changed" from the DecisionSpec
  diff next to "what moved" per task) and --output compare.json share
  every field, and the same inputs and --seed reproduce the record
  exactly. examples/version_compare.py now uses it. (#28)
• Versioned run provenance and comparison compatibility. Every
  EvaluationRunner.run() records a RunProvenance on the batch
  (batch.provenance; provenance in --output and --save-trials JSON;
  a "Run Provenance" section in Markdown and HTML): a measurement side
  (eval-set and per-task SHA-256 content hashes, grader identities with an
  optional declared provenance_version, runner settings) and a
  candidate side (adapter identity, DecisionSpec fingerprint and spec).
  check_compatibility(a, b) returns a CompatibilityReport that is
  compatible, incompatible (changed, added, or removed task content;
  different graders), or unknown (no provenance on a side), with runner
  and version differences as notes and candidate differences reported
  separately with a DecisionSpec diff. Baselines gain task_hash
  (TaskBaseline, update_baseline(task_hash=), promote(task_hash=);
  results carry task_summaries[].task_hash), and the CLI gate refuses to
  compare a task whose content changed since its baseline was stored
  (outcome task_content_changed, gate unevaluable, exit 2) instead of
  matching on id; baselines without a hash still compare, with a warning.
  Checkpoint identity now derives from the same hashing rule (values
  unchanged). Artifacts written before this release load with
  provenance=None; an unknown schema_version is rejected clearly. The
  tracelens init README snippet stores task_hash on each baseline. (#51)
• tracelens run --config tracelens.yaml. A project-owned run
  configuration file holds exactly what the run flags hold (eval set,
  adapter and graders, run counts, outputs, and the baseline gate).
  Precedence is built-in defaults, then the file, then flags given
  explicitly, so an omitted flag never resets a file value, and booleans
  override in both directions (--progress / --no-progress,
  --baseline-check / --no-baseline-check, --require-baselines /
  --no-require-baselines). Paths in the file resolve relative to the
  file; adapters and graders import from run.import_root (default: the
  file's directory) so the command works from any directory; and the file
  is parsed strictly with the safe YAML loader, so unknown keys, duplicate
  keys, wrong types, unsafe constructs, and missing required settings exit
  2 before any agent call. tracelens init now writes tracelens.yaml,
  and the generated README and workflow run the same
  tracelens run --config tracelens.yaml, so enabling the regression gate
  is one edit to the config file. (#35)
• Actionable CLI errors and discoverable outputs. tracelens --debug (or
  TRACELENS_DEBUG=1) adds the full traceback to input and configuration
  errors, which are otherwise one or two lines on stderr with the next
  action; an unimportable adapter or grader now explains the dotted-path
  and project-root requirement. tracelens run validates --num-runs,
  --max-concurrency, --timeout, and --max-infra-retries before doing
  anything, and lists every artifact it wrote on stderr
  ([tracelens] wrote results: ...) while stdout carries only the summary.
  (#48)
• tracelens run accepts JSONL and CSV eval sets. --eval-set picks
  the loader from the file suffix (.json, .jsonl, .csv); a directory
  needs --eval-set-format json|jsonl|csv. --input-field and
  --metadata-fields map foreign JSONL/...

v0.4.0

[0.4.0] - 2026-07-19

Reliability and data-portability release: CI gates now distinguish agent
regressions from harness noise, long runs retry and resume safely, and new
project scaffolding plus JSONL, CSV, and optional Hugging Face loaders shorten
the path from local data to a reproducible evaluation.

Added

• tracelens init. New CLI command that scaffolds a runnable starter
  eval/ suite, including tasks, adapter, grader, README, and a GitHub Actions
  workflow. The command refuses to overwrite generated files unless --force
  is provided.
• Loud CI gate. The baseline check now always prints a gate summary
  (N checked, M skipped (no baseline), K blocking regression(s)), warns
  per task when a baseline is missing, and --require-baselines turns
  missing baselines into a hard failure.
• Configurable infra classification. RunnerConfig.infra_exception_types
  (CLI: --infra-exceptions) extends which exception types are classified
  INFRA_ERROR instead of FAILED. The default set
  (DEFAULT_INFRA_EXCEPTION_TYPES) stays conservative: InfraError,
  MemoryError, ConnectionError.
• Noise-aware gating from the CLI. TaskBaseline.decision_spec stores
  the full spec alongside the fingerprint, --decision-spec loads the
  current run's spec (adapter-stamped transcripts work too), and the
  baseline check now runs compare_with_specs() — so sub-noise-band
  regressions under a mismatched infra config are flagged but not
  blocking, with the infra diff printed. --noise-band tunes the band.
• DecisionSpec write path. update_baseline,
  create_capability_baseline, create_canary_baseline, promote,
  try_promote, and force_promote all accept a decision_spec;
  creation derives the fingerprint from it when one isn't passed, and
  promotion refreshes the stored spec (archiving the old one in
  previous_versions) so it can't drift from the fingerprint.
• DEFAULT_INFRA_EXCEPTION_TYPES is exported top-level (from tracelens import DEFAULT_INFRA_EXCEPTION_TYPES), matching the
  documented + (OSError,) extension pattern.
• Infra-error retry. RunnerConfig.max_infra_retries re-attempts trials
  that end INFRA_ERROR, with exponential backoff
  (infra_retry_backoff_seconds). FAILED and TIMEOUT trials never retry —
  those are observations about the agent, and retrying them would launder
  flakiness out of the pass rate. The final trial records its attempt count in
  Trial.attempts, and retried-away error messages are kept in
  Trial.metadata["infra_retry_errors"]. CLI: --max-infra-retries.
• Checkpoint run identity. Checkpoint files now carry a versioned envelope
  with the eval-set content hash, adapter/grader class identity, and the
  run-level DecisionSpec fingerprint when one is set (class paths alone
  cannot distinguish two configs of the same adapter class). Resuming
  against a checkpoint written by a different eval set, adapter, grader
  stack, or decision spec raises CheckpointError (exported from
  tracelens) instead of silently merging foreign trials keyed only on
  (task_id, run_index). Envelopes with an unknown format version or a
  missing identity are rejected as corrupt. Note: resume requires stable
  explicit task_ids — auto-generated ids change every process.
  Pre-0.4 bare-batch checkpoints still load, with a loud warning that their
  identity can't be verified.
• JSONL and CSV task loaders. JSONLTaskLoader and CSVTaskLoader
  (top-level exports) load eval sets from .jsonl/.csv files or
  directories and save them back, with JSON-compatible round-trips (CSV
  serialises structured Task fields and one canonical metadata column as JSON) and
  no JSON coercion of free-text Task fields. Missing or ambiguous inputs,
  malformed CSV structure, and mixed canonical/flat metadata representations
  fail loudly. The optional HFDatasetLoader loads explicit Hub splits, supports
  revision pinning, and round-trips local saved datasets through the same mapping
  contract without adding a core dependency. Derived from #31 by @Balaji1304.
  Docs: docs/task-sources.md.

Changed

• Gate misconfiguration is now an error. tracelens run --baseline-check without --baselines-file, or with a nonexistent or
  unparseable baselines file, exits 2 before the eval runs instead of
  silently skipping the entire regression check (the file is fully
  loaded during preflight, so a corrupt file can no longer burn a full
  eval before crashing). --require-baselines or --noise-band without
  --baseline-check is also an exit-2 usage error; --baselines-file
  alone warns that it has no effect.
• Harness failures no longer masquerade as agent regressions in the
  gate. The baseline check excludes INFRA_ERROR and grader-crash
  trials from the per-trial comparison samples (they remain visible via
  infra_error_rate / grader_error_rate, a per-task exclusion note,
  and a skipped (no gradable trials) count when nothing gradable
  remains). TIMEOUT trials still count against the agent.
• Adapter-raised TimeoutError is no longer reported as a budget
  timeout. Only the runner's own asyncio.wait_for budget produces
  TrialStatus.TIMEOUT; a TimeoutError from inside the adapter (e.g.
  socket.timeout) now classifies through infra_exception_types
  (FAILED by default, infra if configured) and keeps its original
  message.
• Noise-downgraded reports are internally consistent.
  compare_with_specs() now recomputes overall_severity from the
  blocking regressions and appends a noise-band note to the summary, so
  a noise-only report no longer reads SEVERE while
  should_block_ci() returns False. should_block_ci(..., ignore_noise_band=False) still counts every regression.
• No more fabricated blocking on underpowered zero-variance samples.
  A consistent drop that a valid z-test cannot call significant (e.g.
  five identical scores half a baseline standard deviation below the
  mean) no longer blocks CI — previously it always blocked via the
  fabricated p=0.0. Decisive drops still block; degenerate cases with no
  valid test still block on thresholds with insufficient_data=True.
• No fabricated significance on degenerate samples.
  MetricRegression.p_value is None (not 0.0) when no valid test
  exists — n=1 with baseline_std=0, or zero variance on both sides. Such
  regressions are still reported and can still block CI, with severity
  from the delta thresholds and an explicit insufficient_data flag.
  Zero-variance samples against a known baseline spread now get a real
  z-test.
• Checkpoint resume re-runs infra-errored trials. Resume previously
  skipped every finished trial, permanently freezing INFRA_ERROR results
  into the batch. A rerun with the same checkpoint path now re-executes
  infra-errored trials and SKIPPED placeholders (TIMEOUT trials stay
  skipped — a timeout is an observation about the agent). The checkpoint file format changed to the
  identity envelope described above; old files remain readable.

Fixed

• InfraError docstring matched to behavior. It previously claimed
  OSError and network TimeoutError were classified as infra; they never
  were. The docstring now describes the real (configurable) set and that
  the runner's own budget timeout is always TIMEOUT.
• compare_to_baseline_summary no longer crashes at n=1. The
  Welch-Satterthwaite degrees of freedom fell back to a division by zero
  when either side had a single sample.
• Corrupt checkpoint files fail clearly. An unreadable or unparseable
  checkpoint now raises CheckpointError with the offending path and a
  recovery hint (the CLI prints the error and exits 2 — the misconfigured-run contract) instead of an
  unhandled JSONDecodeError.
• RunnerConfig.fail_fast is honored. The field was previously accepted
  and silently ignored. When enabled, the first trial whose execution fails —
  final status FAILED, INFRA_ERROR (after max_infra_retries is
  exhausted), or TIMEOUT — stops new work from being scheduled. In-flight
  trials still run to completion; unstarted work items produce no trials at
  all, so pass rates, the baseline gate, and checkpoints only ever see
  trials that actually executed (a resume naturally runs the remainder).
  Trials that execute but fail grading, and teardown errors on otherwise
  successful trials, do not trip it. The runner logs how many work items
  were left unrun.

Removed

• Task.max_retries. Dead configuration — the runner never read it.
  Retry policy is an execution concern and lives in
  RunnerConfig.max_infra_retries. Eval-set JSON containing the old field
  still loads; the value is ignored.

v0.3.0

[0.3.0] - 2026-06-10

Hardening release: the grading path now honors its own configuration, harness
failures are first-class signals, and long evaluations survive crashes.

Added

• Grader-crash tracking. Outcome.grader_error marks outcomes synthesized
  from grader crashes; Trial.has_grader_error and
  TrialBatch.grader_error_count/grader_error_rate aggregate them, and
  reports carry the counts next to the existing infra-error stats. A spike
  here means the grading harness broke — not that the agent regressed.
• Checkpoint/resume. RunnerConfig.checkpoint_path and
  checkpoint_interval persist the batch atomically during long runs;
  re-running with the same path resumes, skipping completed trials. CLI:
  --checkpoint.
• Progress reporting. RunnerConfig.progress_callback is called with
  (completed, total) after each trial. CLI: --progress prints per-trial
  progress to stderr.
• DecisionSpec wiring. EvaluationRunner(decision_spec=...) stamps the
  spec onto every transcript that doesn't already carry one, so baselines
  record the reproducibility fingerprint of the run that produced them.
• Token usage roll-up. TrialBatch.total_input_tokens /
  total_output_tokens / total_tokens, mirrored on ReportData, for cost
  visibility without walking every transcript.
• Quality infrastructure. CLI end-to-end integration tests, a Makefile
  with a single make verify gate (lock check → lint → typecheck → tests +
  coverage), and a 90% coverage floor enforced in CI.

Fixed

• LLMGrader honors GraderConfig. Each grading attempt is bounded by
  timeout_seconds, and transient failures — including malformed responses,
  which a fresh LLM call often fixes — retry per retry_on_error /
  max_retries with exponential backoff (new retry_backoff_seconds knob).
  These fields were previously accepted and silently ignored; a hung provider
  stalled the whole eval indefinitely.
• MemoryError from graders propagates (kill-switch) instead of being
  converted into bogus 0-score outcomes for the rest of the run.
• CLI --baseline-check statistics. The regression detector now receives
  one metric sample per trial instead of a single pre-aggregated dict,
  restoring the intended t-test over the sample distribution.

Changed

• Generalized maintainer guidance and public docs for the open source library:
  removed private downstream project references, refreshed CI examples for the
  current CLI, and updated package constraints to the latest PyPI release
  (0.2.0).

v0.2.0

First feature release since the docs-only v0.1.1. Published to PyPI: pip install tracelens==0.2.0.

Highlights

• Human-eval calibration loop — tracelens sample selects trials for human review (diverse / boundary / failures / random) and emits a self-contained worksheet; tracelens reconcile (alias of calibrate) pairs grader vs. human per row (carries trial_id, so no separate results file and multi-run trials stay distinct). Backed by sample_for_review() + CalibrationAnalyzer.analyze_worksheet().
• Infra-noise differentiator — DecisionSpec.InfraConfig, TrialStatus.INFRA_ERROR + InfraError, RegressionDetector.compare_with_specs() (3pp noise band), infra metrics in reports, and a flagship benchmark pack reproducing Anthropic's infra-noise finding.
• Adapters / graders / examples — HTTPAPIAdapter, contract graders, and four runnable examples including human_eval_calibration.py.
• Onboarding docs — human-eval guide, baseline-regression tutorial, evaluation-recipes.

Breaking (0.x)

• Removed WorkflowTask/WorkflowRunner/WorkflowAdapter and LiteLLMProvider; create_provider() supports only "in-memory" (subclass LLMProvider for real vendors).

Full notes: CHANGELOG.md