v0.72.3 — Layer streaming grows up: 6 more architectures, bigger batches, resume, and a disk tier

Layer streaming (v0.72.0–.2) shipped deliberately narrow to prove the mechanism: Llama/Qwen only, batch 1, no gradient accumulation, no resume, RAM only. v0.72.3 removes the training wheels. Every capability was gated against a streamed-vs-resident bit-exactness reference before a line of it was written.

What's New

Six more model families. mistral, gemma, gemma2, gemma3_text, phi and phi3, each verified bit-exact (max abs logit difference 0.0) against the same checkpoint loaded resident — under bf16 and NF4. Phi-3 is the interesting one: it fuses Q/K/V into a single qkv_proj, so there is no q_proj to find, and it is bit-exact anyway. Multimodal gemma3 is deliberately not accepted, only gemma3_text — a real google/gemma-3-* reports gemma3 for its vision wrapper, and streaming that as a causal LM is precisely the silent mis-train the allowlist exists to prevent.

batch_size above 1, and gradient accumulation. Both were refused before. Bigger batches are where streaming pays off: one weight read amortised over more tokens. Measured at equal effective batch, raising batch_size was 2.52x faster than accumulating — accumulation is per-token I/O-neutral, and what it actually buys you is effective batch at constant VRAM. So: raise batch_size until the pre-flight refuses, then accumulate.

A pre-flight that predicts peak VRAM and refuses a run that will not fit. Streaming bounds the weights. It does nothing for activations or for the logits tensor, and both scale with batch x seq. Measured on a 152k-vocab model at batch 8: the logits alone are 8.71 GB — 146x the entire layer-buffer pool. A weights-and-buffers budget waves that configuration straight through. The estimator was fitted to ten real runs across two models, a 3.1x vocabulary contrast, batch 1–8 and two sequence lengths: worst error 0.85%, and it never under-predicts.

A throughput forecast you can trust the provenance of. A range derived from a bf16 GEMM ceiling benchmarked on your card in that session, printed next to the SM clock it was taken at. Never a per-card constant compiled into Soup — the same card here produced 3.5 and 7.6 TFLOPS in two sessions at the same reported clock.

--resume and --hf-resume now work with streaming.

A disk overflow tier. stream_source: auto (the default) uses RAM when the base fits and streams from NVMe when it does not, holding nothing resident. stream_source: ram refuses instead of falling back. Non-NVMe disks are still refused outright — on a spinning disk each step costs two seeks per layer and the run thrashes rather than merely running slower.

soup doctor --disk reports the detected media type.

training:
  stream_layers: true      # base streams out of VRAM; only the adapter trains
  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
  batch_size: 4            # new in v0.72.3
  stream_source: auto      # RAM when it fits, NVMe disk when it does not

Install / Upgrade

pip install -U "soup-cli[train]"

Security

• Subprocess helpers now resolve system tools (powershell, nvidia-smi, diskutil) to absolute paths. On Windows, CreateProcess searches the current directory before PATH, so a bare tool name run from a freshly cloned project could execute an attacker-planted binary sitting in that checkout (CWE-427). No shell metacharacters required.
• A checkpoint carrying two different spellings of the same weight is now refused rather than silently resolved to one of them.
• Streaming weight sources are released when training ends or raises — the disk tier holds one open shard handle per decoder layer.

Fixed

• estimate_logits_bytes charged 6 bytes per logit element; the measured peak is 14 (the loss path holds the bf16 logits, the fp32 upcast, log-softmax's fp32 output and the fp32 gradient live at once). The old figure under-predicted that term by 2.33x.
• Adapters could not be loaded into a streamed model. load_state_dict narrows keys by child name, so a canonical checkpoint matched 0 of N tensors and PEFT emitted only a warning — a resumed run reproduced the from-scratch loss curve byte for byte. Canonical keys are now redirected at load time.
• The NVMe-only tier guard was wired to a hardcoded constant and could never fire.
• The [mcp] extra is now capped at mcp<2. The SDK's 2.0.0 release removed mcp.shared.memory.create_connected_server_and_client_session and dropped Server.list_tools, which breaks soup mcp serve for anyone installing fresh against an unbounded constraint. 1.29.0 (the newest 1.x) was verified to still expose both. Migrating to the 2.x API is tracked separately.

Known Limitations

• The RAM-vs-disk performance gap is unmeasured and no number is claimed. safetensors memory-maps the shards, so the OS page cache keeps them resident between steps on any machine with spare RAM; and at the development box's effective throughput the NVMe read hides under compute. The disk tier's correctness is verified bit-exact against the RAM tier; its speed relative to RAM is not characterised.
• End-to-end soup train --resume could not be demonstrated on the development box, for a reason unrelated to streaming: transformers refuses torch.load below torch 2.6 (CVE-2025-32434), which blocks every resume there — confirmed with a control run using stream_layers: false. The streaming-specific half is verified on the production CUDA path.
• Loading into a streamed model works, but named_parameters() and state_dict() still disagree in memory — the deliberate cost of a serialisation-only design that keeps the earlier bit-exactness guarantees valid.
• Untied embed_tokens + lm_head still stay resident and unquantised.
• The multi-architecture verification used small from-config checkpoints of each family, not large downloaded ones. The module tree comes from the same modelling code, so layer detection and name mapping are exercised identically — but no large Mistral/Gemma/Phi run was performed and none is claimed.
• Layer streaming remains BETA.