Skip to content

Releases: Lsyyh/losslint

losslint 0.4.1

Choose a tag to compare

@Lsyyh Lsyyh released this 25 Aug 12:09

losslint 0.4.1

First PyPI release of the training-log linter.

TUI

  • Sparkline lower edge is now consistently aligned. On Windows the console is switched to the UTF-8 code page so the bottom-aligned block characters render; the non-UTF-8 ASCII fallback ramp was changed so it no longer dips below the baseline.

Packaging

  • SPDX MIT license expression + license-files.
  • Added Changelog / Repository / Documentation project URLs.
  • Zero runtime dependencies; optional [fast] (numpy) and [tb] (TensorBoard) extras.

Install

pip install losslint

Checks

nan_inf, divergence, loss_spike, step_backtrack, overfit_onset, stagnation, parse — with CI-friendly exit codes (0 clean, 1 at/above --fail-severity, 2 usage/parse error).

v0.4.0 — live watch mode, numpy fast path, aligned report

Choose a tag to compare

@Lsyyh Lsyyh released this 25 Aug 11:00

New: losslint watch — live monitoring with evidence snapshots

Point it at the log your trainer is writing right now and it re-lints on
every change, announcing new findings with timestamps. When a finding
reaches the snapshot severity (--snapshot-on, default warning) it
captures an evidence snapshot next to the log — the finding, the tail
of every logged series (lr, grad_norm, ... whatever the trainer
writes), the raw log tail, and host state (load, memory, nvidia-smi
query when present). That is the context you normally lose by the time
you notice the anomaly.

losslint watch runs/exp1/metrics.jsonl --interval 5
  • JSONL / CSV are tailed incrementally — a poll parses only appended
    bytes, so watching a million-step log costs milliseconds per poll
  • trainer_state.json and TensorBoard event files reload in full on change
  • start before the trainer creates the file — the watcher waits
  • --once: single scripted pass (cron / CI against a growing log)
  • --exit-on-finding: exit 1 at --fail-severity for early-stop reactions
  • snapshots are deduplicated per finding and rate-limited per check
    (--snapshot-cooldown, default 60s)

Verified against a real MNIST run: a watcher started before training
caught corrupted-batch loss spikes mid-run and snapshotted lr / grad_norm
/ host state at detection time; a clean baseline stayed silent.

Also in this release

  • pip install losslint[fast] — optional numpy acceleration of the
    checks (finite-value extraction, medians, sliding-window minima, spike
    pre-filter). Stdlib fallbacks return identical results; the package
    stays zero-dependency without the extra. Checks stage on a 500k-point
    log: 0.39s -> 0.20s.
  • Aligned report grid — every series row now renders into the same
    fixed-width grid (framed 24-cell sparkline, right-aligned first -> last
    numbers, real series names), and sparse eval series draw as continuous
    curves instead of blocks scattered across blank cells.
  • CI fix — the 0.3.0 workflow expected the TensorBoard demo (which
    carries a planted NaN by design) to lint clean; it now asserts exit 1.
    Watch and numpy fast-path smoke steps were added.