Skip to content

v0.15.0 — the before-the-GPU release

Choose a tag to compare

@Mormolykos Mormolykos released this 01 Aug 18:10

Every check trainproof shipped until now reads a training log — which means the run already started and the hours are already spent. The failures that cost the most never reach a log at all.

A stack that will not import. A checkpoint that segfaults its own loader. A first batch that exhausts system RAM and freezes the desktop. Zero steps, zero metrics, hours gone — and nothing for a log-based tool to read.

trainproof env checks those before the GPU is touched.

trainproof env --module train --cwd . --checkpoint out/last.ckpt --required-gb 20
[FAIL] TP-ENV-IMPORT-FAIL: 'train' cannot be imported - this run cannot start.
       Evidence: ImportError: cannot import name 'BeamSearchScorer' from
       'transformers'  (raised at .../stream_generator.py:13)
[FAIL] TP-ENV-MEM-INSUFFICIENT: Less system RAM is available than this run declares it needs.
       Evidence: 10.8 GB available, 20.0 GB required (31.1 GB total).

Four check families — stdlib only, no torch, no GPU, no network

Imports run in a subprocess. Not a detail — a safety requirement. The failures here are violent: a segfaulting extension module, a CUDA abort, a library calling os._exit during import. In-process, any of them kills the linter and the user learns nothing. Out of process, a crash with no Python exception is reported as TP-ENV-IMPORT-CRASH and named a native fault — the observable signature of torch.load segfaulting under torch ≥ 2.6 — instead of being misreported as an ImportError.

Checkpoints are never unpickled. torch.load executes arbitrary code by design; that is why torch 2.6 flipped weights_only to True. A linter that must run the file it inspects is not a safety tool. A checkpoint is read as the ZIP archive it is — entry table, tensor-storage count, CRC — distinguishing missing, zero-byte, truncated mid-write, CRC-corrupt, legacy pre-1.6 pickle, and complete. A legacy bare pickle is reported NOT-CHECKED, because refusing to unpickle is correct behaviour, not a defect in the file.

System RAM, not VRAM. A GPU that runs out of memory raises cleanly and the run fails. On Windows the driver spills to system RAM instead, and the machine pages until the desktop stops responding — recoverable only by a hard reset. Where memory cannot be measured the result is TP-ENV-MEM-UNKNOWN; an unmeasurable machine is never reported as a machine with no problems.

Disk. Free space against declared checkpoint size × checkpoints kept.

--cwd

Editable installs and source checkouts resolve relative to the working directory. Probing from anywhere else reports No module named X for a package that imports perfectly where training actually launches — a false FAIL, and the worst kind, because it blames the environment for the linter's own mistake.

Fixed before release

TP-ENV-CKPT-TRUNCATED now fires on an archive with a ZIP header but no central directory — exactly what a save killed mid-write leaves behind. zipfile.is_zipfile() returns False for such a file, so the most common real checkpoint failure was being reported as "not a checkpoint at all". That is the difference between resuming from the previous checkpoint and hunting for a file that was never written. Caught by a test.

Contract

22 new rule IDs, all under TP-ENV-: 62 → 84. schema_version stays 3 — no existing rule, threshold or verdict changed, and no consumer contract is broken. An env run given nothing to judge reports NOT-CHECKED and exits 2 rather than passing vacuously.

Tests: 210 → 228.

pip install trainproof==0.15.0