v0.73.1 — the free GPU tier, and a formula caught under-predicting

A patch release carrying everything that landed since v0.73.0, headed by a fix that made Soup unusable on the cards most people actually have.

What's New

bf16 was assumed on every CUDA card, in fourteen places (#385/#387). Anything pre-Ampere — T4, P100, V100, GTX 16xx, i.e. the entire free tier on Colab and Kaggle — failed on every task, not just streaming. The trap worth knowing: torch.cuda.is_bf16_supported() defaults to including_emulation=True, so a T4 answers True and the first attempt at this fix was a no-op on the exact hardware it was written for.

New: training.stream_vram_probe (#349) decides the layer-streaming VRAM check by measuring one real forward+backward at your configured shape instead of predicting it. The pre-flight's documented contract is that it never under-predicts. Measured through the real soup train on a 4 GB RTX 3050 (SmolLM2-135M streamed bf16, batch 1):

seq	predicted	real peak	ratio
4352	3.282 GB	3.036 GB	1.081x — over-predicts, safe
5120	3.844 GB	4.118 GB	0.934x — under-predicts
6144	4.590 GB	5.830 GB	0.787x — under by 21%

Under-prediction is the direction that does not announce itself: an OOM on Linux, a silent spill to host memory on Windows/WDDM. The grid guarding that contract could not have caught it — all ten of its rows sit at seq 256 or 512, so it varies batch and says nothing about sequence length.

Opt-in, task: sft only, costs 1.0–5.3 s. Gates on max_memory_allocated, not max_memory_reserved (reserved runs 1.08–1.41x allocated, and gating on it would refuse this feature's own flagship 8B NF4 configuration, which runs). Cannot overrule a prediction more than 4x over budget.

Under use_fsdp2_compile, every checkpoint-* still loaded as a dead adapter (#351). The final save was repaired in v0.73.0; the periodic checkpoints were not, so --resume and load_best_model_at_end silently continued from a re-zeroed lora_B. Measured at 70B: 320 canonical keys in the output root, 320 prefixed ones in checkpoint-100.

The MLX adapter_config.json shipped target_modules unresolved, so a default MLX adapter loaded as a silent no-op (#392). _apply_lora resolved target_modules: auto and trained the resolved modules, while the writer serialised the raw "auto". On load linear_to_lora_layers matches no module against that and load_weights(strict=False) drops every LoRA tensor without a word, so generation with the adapter is bit-identical to the base model. auto is the schema default, so this was every MLX run that did not name its modules by hand. Reported by @armanbot-jpg with a control that isolated it: hand-editing keys in the saved file makes the very same adapters.safetensors produce the tuned behaviour — the weights were fine, the config was not.

backend: mlx never dispatched to the MLX trainer (#362/#363), and training.seed reached the SFT wrapper and nothing else (#353) — so a seeded task: grpo run trained at HF's default of 42, and replicates that differed only in their seed were the same run.

Also: training.batch_size now rejects 0 and negatives (-4 parsed before); Colab's preinstalled torchao breaking peft is mapped to a named fix (#389); the dead calibrated_logits_bytes_per_element hook is wired into the pre-flight (#390/#348); training.stream_vram_override (#386); and three published claims retracted where the evidence did not support them, including "bound by host-to-device transfer". No measured number changed in that retraction.

Install / Upgrade

pip install --upgrade "soup-cli[train]"

Security

• training.batch_size accepted 0 and negative values, which reached the streaming VRAM arithmetic that multiplies by it. Now rejected at config load.
• With the measured probe enabled, a predicted miss no longer refuses before the GPU is touched — so the deferral is capped at 4x over budget, since a shared soup.yaml need not be authored by whoever runs it.

Known Limitations

• stream_vram_probe is task: sft only. The four streaming preference losses keep the fitted prediction — which is exactly where it was measured least accurate at long sequence. Measuring a preference step needs the TRL trainer, which does not exist yet at the point the probe runs. #349 stays open for this reason.
• The mechanism of the long-sequence under-prediction is unidentified. seq**2 from the attention score matrix is the obvious candidate and the numbers do not settle it, so the formula is not fixed — only overrulable. Anything that needs the prediction to be right without a GPU (soup profile, plan-only paths) still gets the old accuracy.
• The probe measures a causal-LM step, not the run's own step — +12.5–14.3% conservative against real SFT training, so it refuses slightly earlier than strictly necessary.
• Windows/WDDM still does not raise on an over-budget allocation, so the refusal direction cannot be OOM-verified on that platform; what is validated is the measured demand.
• Layer streaming remains BETA.

Measurement record

benchmarks/gate-v0.73.1-measured-vram-fit.md — published as written, including the three readings withdrawn during the work (two of which briefly looked like the headline result) and one claim that was wrong in draft and is corrected there.