Skip to content

v0.12.0 — the honest-verdict release

Choose a tag to compare

@Mormolykos Mormolykos released this 31 Jul 04:48

No new diagnostic idea. Every change closes a hole in a check that already existed: two silent false negatives, one misdiagnosis, and one claim of coverage the tool had not delivered.

A run whose loss was exactly 0.0 on every step skipped every loss-shape check — each was guarded by a > 0 test to avoid dividing by zero — and reached a PASS whose message then named those same checks as having run.

  • TP-ZERO-LOSS (FAIL): every finite loss is exactly 0.0. Cross-entropy returns 0.0 when every target label is masked to -100, so the finding points at the collator's prompt masking and context-window truncation.
  • TP-ZERO-GRAD (FAIL): every finite gradient norm is exactly 0.0 — a severed backward graph. This was already caught, but as TP-DEAD-RUN ("loss never improved"), which sends you hunting your data and learning rate instead of the graph.
  • TP-CMP-UNCOMPARABLE (FAIL): compare refuses a run with no usable loss scale. A zero loss floor beats any baseline, so such a run used to read as "compares favorably".
  • TP-CMP-ERROR (WARN): doctor --baseline no longer swallows a failed comparison in silence.
  • TP-PASS now reports only checks that actually executed, each skip with its reason, exposed as a new structured checks key. The group list used to be hardcoded.
  • TP-DIVERGE took its floor over all losses, so a single 0.0 anywhere disabled divergence detection for the whole run. It now uses the lowest nonzero loss.
  • The HuggingFace callback dropped every eval entry, which made TP-OVERFIT structurally unreachable there while epoch judged the same data correctly.
  • Six threshold values in RULES.md disagreed with the code and were corrected.

The adapter now preserves trainer_state top-level metadata (max_steps, best_model_checkpoint, …). Inert — no rule reads it at this version.

116 → 166 tests. All 38 golden snapshots byte-identical and EVIDENCE_MATRIX.md differs only in its version stamp, so no existing verdict changed. Full details in CHANGELOG.md.