v0.5.0

TraceLens 0.5.0 makes evaluation results comparable and explainable. Every run records provenance (task content hashes, grader and adapter identity, runner settings) so two runs are checked for compatibility before they are compared; tracelens compare gives a verdict between two saved runs with a paired task bootstrap; tracelens inspect explains failed trials from a trials file; tracelens run --config tracelens.yaml replaces long flag lists; every command shares one exit-code contract; and the pass-rate, pass@k, and pass^k estimators were tightened so harness failures leave the denominator and unevaluable gates no longer pass. Releases are now prepared and published by the release pipeline.

Added

• One-click release preparation. The "Release prepare" workflow takes a
  version, validates it against the tags and the changelog, moves the
  [Unreleased] entries into a dated section (scripts/prepare_release.py,
  which refuses to release nothing), and opens a release: vX.Y.Z pull
  request with the rendered notes; merging it makes the "Release tag"
  workflow tag the merge commit and run the release workflow with
  publish=true. Nothing releases on an ordinary merge, and the manual
  tag path still works. (#88)
• tracelens report --format ci. Re-renders the one-line CI summary
  tracelens run printed, gate line included, from a saved results file,
  so a job summary or script can read it without parsing Markdown. report
  never re-decides the gate: it exits 0. (#75)
• Releases create their GitHub Release automatically. The release
  workflow now runs three jobs: build and verify (tag matches the built
  version; release notes rendered from the changelog's dated section by
  scripts/release_notes.py, failing before anything is published when the
  section is missing or empty), publish to PyPI (skip-existing so a
  re-run of an already-published tag is safe), and create the GitHub
  Release with those notes and the wheel and sdist attached, marked as a
  pre-release for any non-final version. The release step updates an
  existing release instead of failing, so re-runs after a partial failure
  are safe, and workflow_dispatch is a dry run that publishes nothing.
  docs/releasing.md gains the verification checklist and failure/re-run
  guidance. (#54)
• Positioning and contributor guidance aligned with what is actually
  demonstrated. The README and docs home lead with what TraceLens is
  (repo-owned local regression checks, inspectable artifacts, no backend,
  explicit uncertainty) and carry a "What is demonstrated today" table that
  separates the tested mechanism from the not-yet-published downstream
  evidence; the adjacent-tools page acknowledges that hosted platforms also
  run evaluations, experiments, and CI checks and names no vendor features;
  fingerprint explanations no longer claim exact reproducibility or causal
  attribution (DecisionSpec docstrings included); CONTRIBUTING.md
  describes the current extras ([datasets], [docs], PyYAML in core),
  the make verify gate, where things live, and three small first issues
  (#74, #75, #76). (#53)
• The documented user journey runs in CI from a built wheel.
  tests/journey/test_user_journey.py drives real tracelens processes
  through the documented workflow in a scratch project (an existing
  project, init, run --config, baselines from the README snippet, the
  gate enabled in tracelens.yaml, an intentional regression that blocks,
  inspect, compare, a --task-id rerun, an infra outage and a grader
  crash made unevaluable and told apart, malformed input and a bad config,
  checkpoint/resume re-executing nothing, report, sample), checking
  exit codes and persisted decisions at every step. A new CI job installs
  a freshly built wheel into a clean environment and runs the journey
  against its console script (TRACELENS_CLI). (#33, Stage A)
• tracelens inspect: explain failed trials from a trials file.
  tracelens inspect eval/results/trials.json --failures prints, per
  failing trial, one kind (agent failure, infra error, or grader crash,
  never conflated, harness causes first), status and attempts, expected
  versus actual output (--eval-set joins the task's name, input, and
  declared expectation), every grader's verdict, score, metrics, and
  feedback, and the transcript's steps, tokens, tool calls, and errors.
  Absent fields read missing; output is bounded (400 characters per
  field, 20 steps per transcript) with an explicit count of what was
  omitted, and --full lifts the bounds. Filters: --kind agent|infra| grader|not-run|passed, --task-id, --grader (trials that grader
  failed or crashed on), --all, --limit. --html writes a
  self-contained, escaped, offline drilldown that reads on a phone;
  --json writes the same view as data. The command exits 0 whenever the
  file was read (it reports, the gate decides) and 2 on input errors.
  tracelens run --task-id ID ... (run.task_ids in tracelens.yaml)
  reruns only the named tasks and refuses unknown ids; its provenance and
  checkpoint identity cover the subset. New guide: "Debugging a Failed
  Evaluation". (#52)
• tracelens compare: a verdict between two saved runs.
  tracelens compare baseline-trials.json candidate-trials.json (and
  compare_runs() in Python) implements the statistical contract's
  run-versus-run section: tasks are aligned by content through the runs'
  provenance (changed, added, or removed tasks and different graders make
  the runs incompatible; --unmatched-tasks exclude compares the shared
  tasks and lists the rest; artifacts without provenance align by id and
  are labelled, or refused with --require-provenance), one statistic per
  task and run is paired (--metric pass_rate | mean_score | <grader_id>.<metric_name>, --direction lower for latency-like metrics,
  --grader for multi-grader runs), and the mean paired difference gets a
  percentile bootstrap over tasks, a sign-flip p-value (exact for small
  suites), and a verdict against --threshold: improvement, equivalent
  within the threshold, or significant but below it exit 0; regression
  exits 1; inconclusive or insufficient evidence exits 2 (--observe
  forces 0). The terminal summary ("what changed" from the DecisionSpec
  diff next to "what moved" per task) and --output compare.json share
  every field, and the same inputs and --seed reproduce the record
  exactly. examples/version_compare.py now uses it. (#28)
• Versioned run provenance and comparison compatibility. Every
  EvaluationRunner.run() records a RunProvenance on the batch
  (batch.provenance; provenance in --output and --save-trials JSON;
  a "Run Provenance" section in Markdown and HTML): a measurement side
  (eval-set and per-task SHA-256 content hashes, grader identities with an
  optional declared provenance_version, runner settings) and a
  candidate side (adapter identity, DecisionSpec fingerprint and spec).
  check_compatibility(a, b) returns a CompatibilityReport that is
  compatible, incompatible (changed, added, or removed task content;
  different graders), or unknown (no provenance on a side), with runner
  and version differences as notes and candidate differences reported
  separately with a DecisionSpec diff. Baselines gain task_hash
  (TaskBaseline, update_baseline(task_hash=), promote(task_hash=);
  results carry task_summaries[].task_hash), and the CLI gate refuses to
  compare a task whose content changed since its baseline was stored
  (outcome task_content_changed, gate unevaluable, exit 2) instead of
  matching on id; baselines without a hash still compare, with a warning.
  Checkpoint identity now derives from the same hashing rule (values
  unchanged). Artifacts written before this release load with
  provenance=None; an unknown schema_version is rejected clearly. The
  tracelens init README snippet stores task_hash on each baseline. (#51)
• tracelens run --config tracelens.yaml. A project-owned run
  configuration file holds exactly what the run flags hold (eval set,
  adapter and graders, run counts, outputs, and the baseline gate).
  Precedence is built-in defaults, then the file, then flags given
  explicitly, so an omitted flag never resets a file value, and booleans
  override in both directions (--progress / --no-progress,
  --baseline-check / --no-baseline-check, --require-baselines /
  --no-require-baselines). Paths in the file resolve relative to the
  file; adapters and graders import from run.import_root (default: the
  file's directory) so the command works from any directory; and the file
  is parsed strictly with the safe YAML loader, so unknown keys, duplicate
  keys, wrong types, unsafe constructs, and missing required settings exit
  2 before any agent call. tracelens init now writes tracelens.yaml,
  and the generated README and workflow run the same
  tracelens run --config tracelens.yaml, so enabling the regression gate
  is one edit to the config file. (#35)
• Actionable CLI errors and discoverable outputs. tracelens --debug (or
  TRACELENS_DEBUG=1) adds the full traceback to input and configuration
  errors, which are otherwise one or two lines on stderr with the next
  action; an unimportable adapter or grader now explains the dotted-path
  and project-root requirement. tracelens run validates --num-runs,
  --max-concurrency, --timeout, and --max-infra-retries before doing
  anything, and lists every artifact it wrote on stderr
  ([tracelens] wrote results: ...) while stdout carries only the summary.
  (#48)
• tracelens run accepts JSONL and CSV eval sets. --eval-set picks
  the loader from the file suffix (.json, .jsonl, .csv); a directory
  needs --eval-set-format json|jsonl|csv. --input-field and
  --metadata-fields map foreign JSONL/CSV columns the same way the Python
  loaders do. The dispatch is also available as
  tracelens.loaders.load_tasks(), which raises EvalSetLoadError with the
  CLI's message. Hugging Face Hub datasets stay a Python-API concern. (#50)

Changed

• examples/hello_world.py takes --reports-dir. The default is still
  examples/reports/ (the checked-in sample the README links to); the test
  suite now writes to a temporary directory instead of rewriting the sample
  on every run. (#74)
• PyYAML is a core dependency (pyyaml>=6.0), used only through the
  safe loader for --config. Flag-only tracelens run invocations are
  unchanged, except that --eval-set, --adapter, and --graders are now
  required only when no config file provides them; a run missing any of
  them still exits 2, naming both the flag and the config key. (#35)
• The scaffold and the user guide show provenance_version. The
  adapter and grader examples in the user guide declare it with a note on
  when to bump it, and tracelens init writes a commented
  # provenance_version = "starter-1" line into eval/adapter.py and
  eval/grader.py. (#76)
• One exit-code contract for every command. 0 = success or gate passed;
  1 = a negative result (blocked gate, unmet --require-baselines,
  calibration below threshold); 2 = a usage, configuration, or input error,
  or an unevaluable gate. Consequently an unimportable adapter or grader,
  a missing / invalid / non-trials input to tracelens sample, a missing or
  invalid annotations, results, transcripts, or samples file for
  tracelens calibrate (and a worksheet with no usable rows or
  --transcripts without --grader/--samples), and tracelens init
  refusing to overwrite without --force all exit 2 instead of 1 (or a
  traceback). Negative results are unchanged. (#48)
• Eval-set load failures exit 2. A missing path, unsupported suffix,
  directory without --eval-set-format, invalid JSON, malformed record, or
  missing input column now prints a concise error naming the file (and the
  line when the loader knows it) and exits 2 before any agent call;
  previously these exited 1 or raised. (#50)
• Report JSON records the baseline gate decision. tracelens run --output now writes a gate object (status not_requested / passed /
  blocked / unevaluable, exit code, threshold, noise band, task counts,
  reasons, and per-task outcomes with the observed regressions), and
  ReportData.gate carries it in Python. Files written by earlier versions
  have no gate key and load with gate = None; no decision is invented.
• tracelens report and output writing fail clearly. report exits 2
  with a message for a missing results file, invalid JSON, or a document
  that is not a TraceLens results file (ReportData.from_dict now raises
  ValueError instead of rendering an empty report), and tracelens run
  exits 2 when an --output / --report / --html-report /
  --save-trials path cannot be written. Previously both produced
  tracebacks.
• Gate comparisons use gradable trials only. The per-trial samples fed
  to regression detection now follow Trial.is_gradable, so PENDING,
  RUNNING, and SKIPPED trials are excluded like harness failures
  (previously they counted as failures); INFRA_ERROR and grader-crash
  exclusion is unchanged.
• Harness failures and never-run trials leave the pass-rate denominator.
  TrialBatch.pass_rate, passed_count, get_pass_results_by_task(), and
  get_pass_sequences_by_task() now consider only gradable trials
  (Trial.is_gradable: COMPLETED, FAILED, and TIMEOUT without a grader
  crash). INFRA_ERROR trials, grader crashes, and PENDING / RUNNING /
  SKIPPED trials are excluded and appear as None gaps in run sequences;
  the new gradable_count / excluded_count properties and the report's
  gradable_trials field carry the denominator. Suite mean score likewise
  averages gradable trials only. The CLI baseline gate already excluded
  these trials, so gate decisions are unchanged; overall pass rates in
  reports rise for runs that had harness failures. Migration: batches that
  reported pass_rate = passed / total_count now report
  passed / gradable_count; use batch.total_count - batch.gradable_count
  (or ReportData.excluded_trials) to see what was excluded.
• No more c / n fallback for pass@k below k runs.
  pass_at_k_estimator, PassAtKAnalyzer.analyze(), and
  compute_confidence_interval() treat a task with fewer than k gradable
  runs as ineligible, matching pass_to_k_estimator. The float APIs return
  0.0 when no task is eligible (their documented placeholder); use the new
  availability APIs below to tell that apart from a measured zero. Migration:
  code that relied on pass_at_k_estimator({"t": [True]}, k=5) == 1.0
  should either run k trials per task or read
  pass_at_k_metric(...).value is None.
• Report JSON carries availability. ReportData.pass_at_k and
  reliability values (and their per-task counterparts) may be null for
  an unavailable metric, and new top-level keys metric_availability,
  availability_recorded, and gradable_trials (plus gradable_trials
  per task summary) are written. Reports written by earlier versions still
  load: they get availability_recorded = false, their values are shown as
  recorded, and gradable_trials falls back to total_trials.

Fixed

• tracelens init generates a CI workflow that evaluates agent changes
  and installs reproducibly. The generated .github/workflows/eval.yml
  used to trigger only on eval/**, pyproject.toml, and uv.lock, so a
  pull request that changed only agent code skipped the eval; it installed
  with uv pip install tracelens after uv sync, which uv run then
  removed again; it used older action refs than this repository's own CI;
  and its summary step failed on cat when a preflight error meant no
  report was written. It now runs on every pull request to main (a
  paths: filter is shown as an explicit customization), uses
  actions/checkout@v6 and the same pinned astral-sh/setup-uv as this
  repository, installs an existing project from uv.lock with
  uv sync --frozen (or creates an environment in a bare repository) and
  installs TraceLens only when the project does not already provide it,
  pinned to the release that generated the file, runs
  .venv/bin/tracelens directly, tolerates missing report files in the
  summary and artifact steps, and carries the three gate flags as a
  commented block. The generated eval/README.md is a four-step
  walkthrough: run, what CI does, make it yours, and enable the gate with a
  baseline-storing snippet plus a "prove it blocks" step; an end-to-end test
  executes that walkthrough (init, run, store baselines from the README's
  own snippet, break the agent, confirm exit 1 and BLOCKED).
  examples/ci/eval.yml and the CI/CD guide are aligned with the template.
  (#49)
• The gate decision is made once and shown everywhere. tracelens run --baseline-check used to write JSON/Markdown/HTML before comparing against
  baselines, so a run that exited 1 on a severe regression saved artifacts
  that said nothing about it, and re-rendering with tracelens report lost
  regression data entirely. The comparison now runs first
  (tracelens.reporting.gate.evaluate_gate), the resulting GateResult is
  attached to the report before any file is written, and the exit code, the
  stdout summary, the JSON gate object, and the Markdown/HTML "Baseline
  Gate" sections (status, policy, task counts, reasons, a regression table
  with baseline/current/change/severity/notes, skipped tasks) all come from
  it. All-infra and all-grader-error runs stay distinguishable from agent
  regressions in every format, and grader-error rate and token totals are
  now rendered in Markdown, the CI summary, and HTML. (#47)
• Unavailable metrics render as N/A, never as zeros. A one-run-per-task
  suite used to report pass@5 = 1.0 (a fallback) and pass^5 = 0.0 (no
  eligible task) as if measured. New MetricValue
  (tracelens.statistics), pass_at_k_metric, pass_to_k_metric, and
  PassAtKAnalyzer / ConsistencyAnalyzer.analyze_detailed() return the
  value with its evidence: eligible/total task counts, the runs the metric
  needs, the most runs any task recorded, and a reason when unavailable.
  Markdown shows N/A: needs at least 5 gradable runs per task; 0/2 tasks eligible; max 1 gradable run(s) recorded plus a --num-runs hint, the
  CI summary prints pass@5=n/a, HTML lists unavailable metrics under the
  chart instead of drawing zero-height bars, and available values carry
  their eligible/total counts. Pass rates with no gradable trial render as
  N/A in every format, and per-task rows show trials (gradable) when
  they differ. Legacy reports get an explicit note that availability was
  not recorded. (#46)
• pass^k no longer depends on trial completion order. The runner appends
  trials as they finish, and TrialBatch.get_pass_results_by_task() returned
  them in that order, so the consecutive-window pass^k changed with
  concurrency timing and checkpoint resumes: run-index outcomes
  [T, T, F, F] reported pass^2 = 1/3 when trials finished in order and 0
  when they finished in order 0, 2, 3, 1. Results are now ordered by
  run_index. New TrialBatch.get_pass_sequences_by_task() returns
  run-indexed sequences with None for missing runs; pass_to_k,
  pass_to_k_estimator, and ConsistencyAnalyzer accept them, a window
  that would span a gap is not counted, and a task with no complete window
  is ineligible at that k. Reports use these sequences for pass^k. Two
  trials sharing a (task_id, run_index) now raise ValueError instead of
  producing an ambiguous sequence, and pass_to_k rejects k < 1. The
  consistency.py docstring examples that claimed
  pass_to_k([T, T, F, T, T], 3) == 0.333 are corrected to 0.0. pass@k
  is order-invariant and unaffected. (#45)
• pass@k bootstrap intervals preserve repeated task draws.
  PassAtKAnalyzer.compute_confidence_interval and analyze_with_ci
  resampled task IDs with replacement but collected them into a dict keyed
  by task ID, so a task drawn twice counted once: the draw [A, A, B] with
  A=1 and B=0 averaged to 1/2 instead of 2/3. Each resample lost about 37 %
  of its draws and reported intervals were roughly 20–25 % too narrow at
  typical suite sizes; intervals from earlier releases were overconfident
  and should be recomputed. Both methods now compute per-task pass@k once
  (in sorted task_id order) and delegate to bootstrap_ci, and gain an
  optional seed argument: the same inputs and seed give the same interval,
  and input task order no longer affects it. bootstrap_ci, and therefore
  the analyzer, now raises ValueError for confidence outside (0, 1) or
  n_bootstrap < 1 instead of failing inside NumPy. Point estimates and the
  (lower, upper) return shape are unchanged. The estimator, sampling-unit,
  and trial-validity definitions every statistic follows are now written
  down in docs/statistical-contract.md. (#44)
• Unevaluable baseline gates no longer pass. tracelens run --baseline-check exits 2 when any baseline-backed task has no gradable
  trials or no comparable CLI metrics, or when no task can be checked at all
  (including empty suites, zero runs, and empty/unrelated baseline files).
  The summary marks the gate UNEVALUABLE and retains observed regressions
  and exclusion counts. This intentionally changes the previous exit-0
  behavior: infrastructure and grader failures remain excluded from agent
  regression samples, but missing evidence cannot authorize a passing gate.
  An unevaluable gate takes precedence over exit 1 for policy violations;
  otherwise regression and required-baseline failures still exit 1. Partial
  trial loss remains allowed when every baseline-backed task retains a
  gradable sample. Non-gated runs are unchanged.

Install: pip install tracelens==0.5.0 · PyPI · Full changelog