v0.4.0

[0.4.0] - 2026-07-19

Reliability and data-portability release: CI gates now distinguish agent
regressions from harness noise, long runs retry and resume safely, and new
project scaffolding plus JSONL, CSV, and optional Hugging Face loaders shorten
the path from local data to a reproducible evaluation.

Added

• tracelens init. New CLI command that scaffolds a runnable starter
  eval/ suite, including tasks, adapter, grader, README, and a GitHub Actions
  workflow. The command refuses to overwrite generated files unless --force
  is provided.
• Loud CI gate. The baseline check now always prints a gate summary
  (N checked, M skipped (no baseline), K blocking regression(s)), warns
  per task when a baseline is missing, and --require-baselines turns
  missing baselines into a hard failure.
• Configurable infra classification. RunnerConfig.infra_exception_types
  (CLI: --infra-exceptions) extends which exception types are classified
  INFRA_ERROR instead of FAILED. The default set
  (DEFAULT_INFRA_EXCEPTION_TYPES) stays conservative: InfraError,
  MemoryError, ConnectionError.
• Noise-aware gating from the CLI. TaskBaseline.decision_spec stores
  the full spec alongside the fingerprint, --decision-spec loads the
  current run's spec (adapter-stamped transcripts work too), and the
  baseline check now runs compare_with_specs() — so sub-noise-band
  regressions under a mismatched infra config are flagged but not
  blocking, with the infra diff printed. --noise-band tunes the band.
• DecisionSpec write path. update_baseline,
  create_capability_baseline, create_canary_baseline, promote,
  try_promote, and force_promote all accept a decision_spec;
  creation derives the fingerprint from it when one isn't passed, and
  promotion refreshes the stored spec (archiving the old one in
  previous_versions) so it can't drift from the fingerprint.
• DEFAULT_INFRA_EXCEPTION_TYPES is exported top-level (from tracelens import DEFAULT_INFRA_EXCEPTION_TYPES), matching the
  documented + (OSError,) extension pattern.
• Infra-error retry. RunnerConfig.max_infra_retries re-attempts trials
  that end INFRA_ERROR, with exponential backoff
  (infra_retry_backoff_seconds). FAILED and TIMEOUT trials never retry —
  those are observations about the agent, and retrying them would launder
  flakiness out of the pass rate. The final trial records its attempt count in
  Trial.attempts, and retried-away error messages are kept in
  Trial.metadata["infra_retry_errors"]. CLI: --max-infra-retries.
• Checkpoint run identity. Checkpoint files now carry a versioned envelope
  with the eval-set content hash, adapter/grader class identity, and the
  run-level DecisionSpec fingerprint when one is set (class paths alone
  cannot distinguish two configs of the same adapter class). Resuming
  against a checkpoint written by a different eval set, adapter, grader
  stack, or decision spec raises CheckpointError (exported from
  tracelens) instead of silently merging foreign trials keyed only on
  (task_id, run_index). Envelopes with an unknown format version or a
  missing identity are rejected as corrupt. Note: resume requires stable
  explicit task_ids — auto-generated ids change every process.
  Pre-0.4 bare-batch checkpoints still load, with a loud warning that their
  identity can't be verified.
• JSONL and CSV task loaders. JSONLTaskLoader and CSVTaskLoader
  (top-level exports) load eval sets from .jsonl/.csv files or
  directories and save them back, with JSON-compatible round-trips (CSV
  serialises structured Task fields and one canonical metadata column as JSON) and
  no JSON coercion of free-text Task fields. Missing or ambiguous inputs,
  malformed CSV structure, and mixed canonical/flat metadata representations
  fail loudly. The optional HFDatasetLoader loads explicit Hub splits, supports
  revision pinning, and round-trips local saved datasets through the same mapping
  contract without adding a core dependency. Derived from #31 by @Balaji1304.
  Docs: docs/task-sources.md.

Changed

• Gate misconfiguration is now an error. tracelens run --baseline-check without --baselines-file, or with a nonexistent or
  unparseable baselines file, exits 2 before the eval runs instead of
  silently skipping the entire regression check (the file is fully
  loaded during preflight, so a corrupt file can no longer burn a full
  eval before crashing). --require-baselines or --noise-band without
  --baseline-check is also an exit-2 usage error; --baselines-file
  alone warns that it has no effect.
• Harness failures no longer masquerade as agent regressions in the
  gate. The baseline check excludes INFRA_ERROR and grader-crash
  trials from the per-trial comparison samples (they remain visible via
  infra_error_rate / grader_error_rate, a per-task exclusion note,
  and a skipped (no gradable trials) count when nothing gradable
  remains). TIMEOUT trials still count against the agent.
• Adapter-raised TimeoutError is no longer reported as a budget
  timeout. Only the runner's own asyncio.wait_for budget produces
  TrialStatus.TIMEOUT; a TimeoutError from inside the adapter (e.g.
  socket.timeout) now classifies through infra_exception_types
  (FAILED by default, infra if configured) and keeps its original
  message.
• Noise-downgraded reports are internally consistent.
  compare_with_specs() now recomputes overall_severity from the
  blocking regressions and appends a noise-band note to the summary, so
  a noise-only report no longer reads SEVERE while
  should_block_ci() returns False. should_block_ci(..., ignore_noise_band=False) still counts every regression.
• No more fabricated blocking on underpowered zero-variance samples.
  A consistent drop that a valid z-test cannot call significant (e.g.
  five identical scores half a baseline standard deviation below the
  mean) no longer blocks CI — previously it always blocked via the
  fabricated p=0.0. Decisive drops still block; degenerate cases with no
  valid test still block on thresholds with insufficient_data=True.
• No fabricated significance on degenerate samples.
  MetricRegression.p_value is None (not 0.0) when no valid test
  exists — n=1 with baseline_std=0, or zero variance on both sides. Such
  regressions are still reported and can still block CI, with severity
  from the delta thresholds and an explicit insufficient_data flag.
  Zero-variance samples against a known baseline spread now get a real
  z-test.
• Checkpoint resume re-runs infra-errored trials. Resume previously
  skipped every finished trial, permanently freezing INFRA_ERROR results
  into the batch. A rerun with the same checkpoint path now re-executes
  infra-errored trials and SKIPPED placeholders (TIMEOUT trials stay
  skipped — a timeout is an observation about the agent). The checkpoint file format changed to the
  identity envelope described above; old files remain readable.

Fixed

• InfraError docstring matched to behavior. It previously claimed
  OSError and network TimeoutError were classified as infra; they never
  were. The docstring now describes the real (configurable) set and that
  the runner's own budget timeout is always TIMEOUT.
• compare_to_baseline_summary no longer crashes at n=1. The
  Welch-Satterthwaite degrees of freedom fell back to a division by zero
  when either side had a single sample.
• Corrupt checkpoint files fail clearly. An unreadable or unparseable
  checkpoint now raises CheckpointError with the offending path and a
  recovery hint (the CLI prints the error and exits 2 — the misconfigured-run contract) instead of an
  unhandled JSONDecodeError.
• RunnerConfig.fail_fast is honored. The field was previously accepted
  and silently ignored. When enabled, the first trial whose execution fails —
  final status FAILED, INFRA_ERROR (after max_infra_retries is
  exhausted), or TIMEOUT — stops new work from being scheduled. In-flight
  trials still run to completion; unstarted work items produce no trials at
  all, so pass rates, the baseline gate, and checkpoints only ever see
  trials that actually executed (a resume naturally runs the remainder).
  Trials that execute but fail grading, and teardown errors on otherwise
  successful trials, do not trip it. The runner logs how many work items
  were left unrun.

Removed

• Task.max_retries. Dead configuration — the runner never read it.
  Retry policy is an execution concern and lives in
  RunnerConfig.max_infra_retries. Eval-set JSON containing the old field
  still loads; the value is ignored.