Skip to content

v0.9.0 — the eval-aware release

Choose a tag to compare

@Mormolykos Mormolykos released this 20 Jul 02:23

trainproof learns to read the eval curve — and a real bug the evidence exposed.

Fix: false TP-DIVERGE on steeply-converging runs

HuggingFace trainer_state.json ends with a training-summary entry whose train_loss is the run average, not a per-step loss. It was leaking into the per-step loss series, and on any run that converges steeply (high start, low end) it could fire a false TP-DIVERGE. The HF adapter now drops that summary entry. Caught by the overfit evidence run below — evidence-driven development doing its job.

New rule: TP-OVERFIT (WARN)

Deterministic overfitting detection: eval_loss rising past 1.2x of its own minimum while train_loss keeps falling (needs >= 4 eval points). It fires WARN, never FAIL — early stopping is your choice; the rule flags that your best checkpoint was earlier. Documented in RULES.md.

Grounded in a real Qwen2.5-3B QLoRA overfit run now shipped in examples/gallery/overfit/: eval_loss bottomed at 1.25 (step 30) and climbed to 3.76 (step 300) while train_loss fell 1.38 -> 0.03.

Unchanged

No existing rule thresholds changed; the five original gallery verdicts are identical. 57 tests passing.