v0.17.0 — the lint gate, and one log that used to vanish

Lint had never been part of the gate. No [tool.ruff] section, ruff not a dependency, 103 findings against inherited defaults. A repository that ships a linter should not fail its own — it now reports zero.

The defect the gate surfaced

doctor walks a directory twice: once to discover candidate logs, once to judge them. The judging pass already reported what it could not read — "could not be parsed and were NOT judged".

The discovery pass did not. A file that raised there was swallowed by except Exception: pass, never became a candidate, and so never reached that note.

The result: a file plainly visible on disk, absent from the report, and indistinguishable from one that passed. That is precisely the failure NOT-CHECKED was introduced to prevent, sitting one loop earlier than anyone had looked.

Both passes now feed the same note. The regression test was verified to fail without the fix before it was kept — a test that passes either way proves nothing.

Two other broad handlers were narrowed, not removed. A non-JSON line inside a JSONL log, and a {-prefixed line that is not valid JSON, are expected inputs and skipping them is correct — but they now catch json.JSONDecodeError specifically, so a genuine fault in the surrounding code can no longer disguise itself as an unparseable line.

The ruleset is chosen, not inherited

E, F, I, B, plus RUF013 (an implicit Optional is a type hint that lies), RUF059, and S110 (try-except-pass).

Adopting everything ruff reports would make this codebase worse, because several of its opinions contradict deliberate decisions. Every exclusion carries its reason in pyproject.toml:

• BLE001 (blind except) — trainproof catches broad exceptions on purpose when parsing logs it did not write and probing subprocesses that can die in ways Python cannot describe. A narrow except there would let an unforeseen parser error escape as a traceback instead of exit code 2, "cannot judge". S110 is enforced instead: catching broadly is fine, catching and passing is not.
• PLW1510 (subprocess without check) — every subprocess call inspects returncode itself and turns it into a finding. check=True would raise, which is the opposite of the required behaviour.
• E501 (line length) — most long lines are the evidence strings trainproof prints, asserted in tests and encoded byte-for-byte in the golden snapshots. Reflowing them would risk changing the tool's output to satisfy a ruler.

Verification

• Zero ruff findings, enforced in the release ritual
• 228 → 230 tests
• All 38 golden snapshots byte-identical
• scripts/regenerate_evidence.py --check exits 0

A cleanup that moves a verdict is not a cleanup.

pip install trainproof==0.17.0

v0.16.0 — the rule registry (no behaviour change)

Every single-run rule lived inside one function, check_records() — which also computed the statistics all the other rules depended on. Adding a check meant editing several hundred lines that every existing check ran through, so each new rule raised the risk to the rules already there. That was the structural bottleneck, and it was blocking the checkpoint work planned next.

What changed

Each rule is now a standalone function taking a CheckContext and returning findings. check_records() builds the context and runs a registry in order — composition instead of implementation.

The context computes each shared series exactly once — losses, gradient norms, learning rates, step times, eval losses, loader fractions — and carries the ran / skipped bookkeeping. A rule can now be read, tested and reasoned about without reading the ones around it.

What did not change — and how that is proven

No rule, threshold, verdict, evidence string or output shape moved. That is enforced, not asserted:

• All 38 golden snapshots byte-identical to 0.15.0
• scripts/regenerate_evidence.py --check exits 0 — the evidence matrix regenerates unchanged
• The same 228 tests pass, none of them modified

Rule evaluation order is part of that guarantee. The snapshots encode the sequence findings appear in, so reordering the registry would change them even if every individual verdict stayed identical. That is what makes the byte-identical comparison a real test rather than a formality.

A refactor of judging logic that cannot prove it changed nothing is indistinguishable from a silent regression. This release exists to be provable.

Shipped alone, on purpose

Mixing a behaviour-preserving refactor with a new check would destroy the only evidence that the refactor preserved behaviour. The checkpoint-tensor work lands in its own release, against a codebase where adding a rule no longer means editing every other one.

84 rule IDs, 228 tests, schema_version 3 — all unchanged.

pip install trainproof==0.16.0

v0.15.0 — the before-the-GPU release

Every check trainproof shipped until now reads a training log — which means the run already started and the hours are already spent. The failures that cost the most never reach a log at all.

A stack that will not import. A checkpoint that segfaults its own loader. A first batch that exhausts system RAM and freezes the desktop. Zero steps, zero metrics, hours gone — and nothing for a log-based tool to read.

trainproof env checks those before the GPU is touched.

trainproof env --module train --cwd . --checkpoint out/last.ckpt --required-gb 20

[FAIL] TP-ENV-IMPORT-FAIL: 'train' cannot be imported - this run cannot start.
       Evidence: ImportError: cannot import name 'BeamSearchScorer' from
       'transformers'  (raised at .../stream_generator.py:13)
[FAIL] TP-ENV-MEM-INSUFFICIENT: Less system RAM is available than this run declares it needs.
       Evidence: 10.8 GB available, 20.0 GB required (31.1 GB total).

Four check families — stdlib only, no torch, no GPU, no network

Imports run in a subprocess. Not a detail — a safety requirement. The failures here are violent: a segfaulting extension module, a CUDA abort, a library calling os._exit during import. In-process, any of them kills the linter and the user learns nothing. Out of process, a crash with no Python exception is reported as TP-ENV-IMPORT-CRASH and named a native fault — the observable signature of torch.load segfaulting under torch ≥ 2.6 — instead of being misreported as an ImportError.

Checkpoints are never unpickled. torch.load executes arbitrary code by design; that is why torch 2.6 flipped weights_only to True. A linter that must run the file it inspects is not a safety tool. A checkpoint is read as the ZIP archive it is — entry table, tensor-storage count, CRC — distinguishing missing, zero-byte, truncated mid-write, CRC-corrupt, legacy pre-1.6 pickle, and complete. A legacy bare pickle is reported NOT-CHECKED, because refusing to unpickle is correct behaviour, not a defect in the file.

System RAM, not VRAM. A GPU that runs out of memory raises cleanly and the run fails. On Windows the driver spills to system RAM instead, and the machine pages until the desktop stops responding — recoverable only by a hard reset. Where memory cannot be measured the result is TP-ENV-MEM-UNKNOWN; an unmeasurable machine is never reported as a machine with no problems.

Disk. Free space against declared checkpoint size × checkpoints kept.

--cwd

Editable installs and source checkouts resolve relative to the working directory. Probing from anywhere else reports No module named X for a package that imports perfectly where training actually launches — a false FAIL, and the worst kind, because it blames the environment for the linter's own mistake.

Fixed before release

TP-ENV-CKPT-TRUNCATED now fires on an archive with a ZIP header but no central directory — exactly what a save killed mid-write leaves behind. zipfile.is_zipfile() returns False for such a file, so the most common real checkpoint failure was being reported as "not a checkpoint at all". That is the difference between resuming from the previous checkpoint and hunting for a file that was never written. Caught by a test.

Contract

22 new rule IDs, all under TP-ENV-: 62 → 84. schema_version stays 3 — no existing rule, threshold or verdict changed, and no consumer contract is broken. An env run given nothing to judge reports NOT-CHECKED and exits 2 rather than passing vacuously.

Tests: 210 → 228.

pip install trainproof==0.15.0

v0.14.0 — the third-framework release

Every rule shipped so far had only ever been tested against HuggingFace trainer_state.json. PyTorch Lightning, Fish Speech and most research code write their metrics to TensorBoard event files and nowhere else — so those runs were invisible. A Lightning run could overfit for three hours and trainproof had nothing to read.

Added

A fifth log format: tfevents. A TensorBoard event-file reader written from the wire format — TFRecord framing plus the Event / Summary / TensorProto fields it needs.

It imports no tensorflow, no tensorboard, no protobuf, no torch, no numpy. Reading a log file should not require installing a training stack, and trainproof's zero-dependency guarantee is unchanged.

Validated against the reference implementation rather than against itself: on a real 2049-step Lightning run it reproduces tensorboard's own EventAccumulator byte-exact — all 13 tags, every point count, first and last values to 1e-6.

• Rank-0 tensor scalars decoded alongside simple_value (Lightning writes the former; a reader handling only the latter sees an empty run)
• Cross-framework tag normalisation — train/loss, TrainIterStats/loss, training/loss → loss; val/loss, EvalStats/avg_loss → eval_loss; lr-AdamW/pg1 → lr
• When several tags claim one column, the denser series wins, ties alphabetical — deterministic, not dict-order dependent
• --format tfevents on epoch, doctor, compare, watch; directories are scanned and shards merged
• Truncated files — the normal state of a killed run — are read up to the cut

Fixed

TP-ZERO-GRAD false positive. The rule condemned any run whose finite gradient norms were all exactly 0.0, reporting a severed backward graph. Coqui writes avg_grad_norm as 0.0 when clipping is off, so a healthy 125,000-step XTTS fine-tune that reached loss 0.017 was reported FAIL.

A run cannot both learn and receive no gradient. The check now stands down when the loss improved, recording the reason as a skip, and stays armed when the loss is stuck.

Found by running the shipped rules against a real training run — not by a test.

Evidence

evidence/ now ships the logs behind these claims: a 125,000-step Coqui XTTS v2 fine-tune as both a text log and the event file from the same run, and a 2049-step Fish Speech LoRA fine-tune. Both on an RTX 5080.

Judged by the shipped rules at generation time:

framework	format	records	verdict
Coqui XTTS v2	coqui	2501	FAIL TP-DIVERGE, TP-THROUGHPUT
Coqui XTTS v2	tfevents	1255	FAIL TP-DIVERGE, TP-THROUGHPUT
Lightning / Fish Speech	tfevents	82	WARN TP-OVERFIT, TP-THROUGHPUT

The two XTTS rows are the same run read by completely independent parsers. They agree exactly — and EVIDENCE_MATRIX.md now computes that agreement as a derived observation rather than asserting it in prose.

Contract

No new rule IDs (62, unchanged) and no schema change (schema_version 3). 0.14.0 adds a capability, it does not break one.

Tests: 179 → 210.

pip install trainproof==0.14.0
trainproof doctor results/my_run/tensorboard/version_0

v0.13.0 — the third-state release

A third verdict, and no new detection.

Before this release, a run where no check group could execute returned PASS with exit 0 — so "checked and clean" and "nothing could be checked" were indistinguishable to the only surface CI reads.

• NOT-CHECKED (exit 2) fires when zero check groups executed. In practice that means fewer than five finite loss points and a non-positive mean loss — a short run whose loss is all zeros, the sub-threshold companion to TP-ZERO-LOSS. A short log with positive losses is not NOT-CHECKED: divergence and flat-loss are guarded by loss positivity rather than point count, so they run and the verdict is an honest PASS.
• Severity and exit code are now two separate axes. Severity orders FAIL > WARN > NOT-CHECKED > PASS. Exit is 1 for any FAIL, else 2 for anything unjudged, else 0.
• An unrecognised verdict normalises to NOT-CHECKED and exits 2. It previously ranked below PASS and matched neither exit branch, so a report trainproof could not classify was reported to CI as clean.
• All four renderers updated together — console, the doctor counts line, SARIF (mapped to warning, not note), and the HTML report.
• schema_version 2 → 3. The verdict enum gained a member, which is breaking for any consumer switching on verdict — hence a minor bump, not a patch.

166 → 179 tests. All 38 golden snapshots byte-identical and EVIDENCE_MATRIX.md differs only in its version stamp, so no existing verdict changed. Full details in CHANGELOG.md.

v0.12.0 — the honest-verdict release

No new diagnostic idea. Every change closes a hole in a check that already existed: two silent false negatives, one misdiagnosis, and one claim of coverage the tool had not delivered.

A run whose loss was exactly 0.0 on every step skipped every loss-shape check — each was guarded by a > 0 test to avoid dividing by zero — and reached a PASS whose message then named those same checks as having run.

• TP-ZERO-LOSS (FAIL): every finite loss is exactly 0.0. Cross-entropy returns 0.0 when every target label is masked to -100, so the finding points at the collator's prompt masking and context-window truncation.
• TP-ZERO-GRAD (FAIL): every finite gradient norm is exactly 0.0 — a severed backward graph. This was already caught, but as TP-DEAD-RUN ("loss never improved"), which sends you hunting your data and learning rate instead of the graph.
• TP-CMP-UNCOMPARABLE (FAIL): compare refuses a run with no usable loss scale. A zero loss floor beats any baseline, so such a run used to read as "compares favorably".
• TP-CMP-ERROR (WARN): doctor --baseline no longer swallows a failed comparison in silence.
• TP-PASS now reports only checks that actually executed, each skip with its reason, exposed as a new structured checks key. The group list used to be hardcoded.
• TP-DIVERGE took its floor over all losses, so a single 0.0 anywhere disabled divergence detection for the whole run. It now uses the lowest nonzero loss.
• The HuggingFace callback dropped every eval entry, which made TP-OVERFIT structurally unreachable there while epoch judged the same data correctly.
• Six threshold values in RULES.md disagreed with the code and were corrected.

The adapter now preserves trainer_state top-level metadata (max_steps, best_model_checkpoint, …). Inert — no rule reads it at this version.

116 → 166 tests. All 38 golden snapshots byte-identical and EVIDENCE_MATRIX.md differs only in its version stamp, so no existing verdict changed. Full details in CHANGELOG.md.

v0.11.0 — the evidence release

Evidence release. No rule and no threshold changed — every verdict from
0.10.0 is byte-identical in tests/golden/.

• All three seeds of every gallery configuration now ship: 18 runs
  (6 configs x seeds 42/43/44). Previously only one seed per config was
  committed, so "3 seeds out of 3" is now a claim you can check.
• New: examples/real_world/xtts_diverged — a 9.8-hour Coqui XTTS fine-tune
  that diverged on its own. The first shipped failure nobody injected, and
  the only Coqui-format fixture.
• EVIDENCE_MATRIX.md is now generated from the logs by
  scripts/regenerate_evidence.py, version-stamped, with the rule IDs that
  fired in each cell. A test fails the build if it goes stale.
• python -m trainproof now works.
• The HTML report is opt-in via epoch --html [PATH] instead of being
  written into your working directory on every run.
• compare now labels table rows with enough path to tell them apart.

116 tests. Full details in CHANGELOG.md.

v0.10.0 — the contract release

Nothing about how trainproof judges a run changed in this release. No rule, no threshold, and no verdict moved — every gallery verdict is locked in tests/golden/ and byte-identical to v0.9. What changed is what trainproof promises, now written down in CONTRACTS.md.

New: CONTRACTS.md

An explicit statement of what this tool guarantees: exit codes, JSON schema policy, rule-ID stability, SARIF mapping, the verdict-stability guarantee, and the pre-1.0 breaking-change policy.

Writing it found a real bug that had already shipped. See below.

New: SARIF 2.1.0 output

--sarif PATH on data, tokenizer, epoch, doctor, compare and preflight. Findings become GitHub PR annotations, so a doomed fine-tune is flagged inline on the diff that caused it instead of buried in a CI log nobody opens. Works independently of --json.

--json is now available on data, tokenizer and preflight as well (previously epoch, doctor and compare only).

Breaking: exit codes

2 now means "trainproof could not judge" — unreadable log, missing file, no parsed records, missing optional dependency. Several of these previously exited 1, which is reserved for a FAIL verdict about your run.

CI that treats any non-zero code as failure is unaffected. Anything that distinguishes 1 from 2 should be reviewed.

An unreadable log no longer reports worst_verdict: "FAIL" in JSON. It reports worst_verdict: null with a populated error key, and exits 2.

Breaking: schema_version is now 2

doctor no longer emits a separate compare_findings key. Single-run and baseline findings live in one findings array, each tagged with source (single_run or compare). The envelope gained an error key.

Fixed: doctor --baseline printed FAIL and exited 0

The exit code was computed from single-run verdicts alone and never consulted the comparison, so doctor --baseline could print [FAIL] comparison findings and still exit 0. The printed output had been telling the truth; the exit code had been lying. bad_labels against healthy is the reproducing case.

Found by the contract work — writing down which of the two signals was authoritative is what exposed that no test had ever checked they agreed.

Also fixed:

• Uncaught internal errors previously fell through to Python's default exit code 1 and were indistinguishable from a FAIL verdict. A top-level handler now reports them as 2.
• A missing transformers install is no longer a FAIL verdict on your dataset (rule TP-PRE-TRANSFORMERS-MISSING removed); it is a tool error, exit 2.
• "Cannot judge" messages now go to stderr, leaving stdout parseable.
• RULES.md no longer carries a stale version stamp.
• The v0.5.0 entry in the changelog described a coroner command that was never implemented; corrected to epoch.

Regression locking

Gallery snapshots in tests/golden/ now pin the verdict and the complete rule ID set for all six runs and seven baseline comparisons. A rule that stops firing and one that starts firing spuriously both fail the build.

Two new tests enforce that rule IDs in the source and in RULES.md match in both directions, and that the two declared version strings agree.

---

85 tests passing. 57 stable rule IDs. pip install trainproof

v0.9.0 — the eval-aware release

trainproof learns to read the eval curve — and a real bug the evidence exposed.

Fix: false TP-DIVERGE on steeply-converging runs

HuggingFace trainer_state.json ends with a training-summary entry whose train_loss is the run average, not a per-step loss. It was leaking into the per-step loss series, and on any run that converges steeply (high start, low end) it could fire a false TP-DIVERGE. The HF adapter now drops that summary entry. Caught by the overfit evidence run below — evidence-driven development doing its job.

New rule: TP-OVERFIT (WARN)

Deterministic overfitting detection: eval_loss rising past 1.2x of its own minimum while train_loss keeps falling (needs >= 4 eval points). It fires WARN, never FAIL — early stopping is your choice; the rule flags that your best checkpoint was earlier. Documented in RULES.md.

Grounded in a real Qwen2.5-3B QLoRA overfit run now shipped in examples/gallery/overfit/: eval_loss bottomed at 1.25 (step 30) and climbed to 3.76 (step 300) while train_loss fell 1.38 -> 0.03.

Unchanged

No existing rule thresholds changed; the five original gallery verdicts are identical. 57 tests passing.

v0.8.0 — the trust release

Never judge the wrong number, never be un-greppable.

Canonical record contract

Adapters are now the ONLY place log columns map to canonical keys — exact-name matching, never substring. eval_loss can no longer be mistaken for training loss; unknown columns (e.g. grad_accum) are dropped instead of misjudged. --map CANON=COLUMN overrides the table when your logs are weird, and doctor prints column provenance for generic logs (COLUMNS: loss<-'train_loss').

Stable rule IDs

Every finding now carries a permanent ID — TP-DIVERGE, TP-DEAD-RUN, TP-ZERO-LR, TP-STEP-CLIFF, … — documented in RULES.md (what fires it, the threshold, and what it does NOT mean).

⚠️ BREAKING for text-parsers: output lines now include the ID ([FAIL] TP-DIVERGE: ...). Parse --json instead — that is what it is for. Preflight IDs renamed to TP-PRE-*.

Honest PASS

TP-PASS now states which check groups actually ran and which were skipped for lack of data: "Ran: loss-shape, divergence, dead-run, grad-norm, lr. Skipped (no data): timing." The tool no longer implies stability of anything it did not measure.

Machine-readable output

--json on epoch / doctor / compare: one JSON document — schema_version: 1, trainproof_version, full reports with rule IDs, worst verdict. Exit codes unchanged.

Also

• HF adapter now captures eval_loss records (no rule uses them yet — that is the next release).
• New README section: For AI coding agents — if your agent checks training projects, trainproof doctor . --json is built for it.
• 52 tests passing. No rule threshold changes; all gallery verdicts identical to v0.7.