v0.73.0 — Three days on somebody else's hardware

Every number this project had ever published was measured on one machine: an RTX 3050 Laptop, 4 GB, Windows. From 5–9 August it ran on a borrowed 8×H100 box (Ubuntu 24.04, a much newer torch / bitsandbytes / trl / peft stack) for the first time.

That found one silent correctness defect in layer streaming, four backends that had never actually run, and a documented multi-GPU entry point that had never launched — and it confirmed the headline claim on hardware nothing like the one it was made on.

The full measurement record, published as written including six rejected hypotheses and three false positives that controls caught, is benchmarks/gate-h100-validation.md.

This is a minor bump, not a v0.72.x patch: it adds two capabilities that did not exist and repairs four backends.

What's New

The laptop result reproduces on completely different hardware

Llama-3.1-8B NF4, streamed: 119.6 tok/s in a 3.32 GB peak on the RTX 3050, against a median 113.00 tok/s in the same 3.32 GB on an H100. Layer streaming is bound by host-to-device transfer, not by the GPU — the first evidence of that from outside the original box.

A silent wrong-gradient defect, found and repaired (#331)

On NF4 models above ~165 MiB per layer (32B and up), bitsandbytes.MatMul4Bit kept the packed weight and quant_state on ctx as plain attributes instead of through save_for_backward, so gradient checkpointing could not discard and recompute them. The reference aliased the streaming buffer pool and was read after that slot had been refilled — a bit-exact forward, a healthy-looking loss curve, and wrong gradients on every layer but the last stream_buffers.

Repaired by keeping the weight out of that function entirely (dequantise inside the checkpointed region). Gated against a resident NF4 reference with a repair-disabled control in the same process:

Model	Gradients exact	Control	Peak VRAM	Throughput
Real 32B	256/256	8–12/256	+2.9%	−4.8%
Real 72B	320/320	8/320	+2.6%	−3.7%

Threshold bracketed at 163.8–171.5 MiB/layer; never bf16, never 8B or 14B. De-aliasing was measured and rejected first — bnb holds the reference across the whole forward-to-backward span, so any copy is O(model), taking real 32B from 4,220 to 19,720 MiB.

Four backends that had never actually run, now do

• soup train --gpus N never launched (#77). accelerate launch takes a script path positionally and Soup handed it sys.executable, so accelerate parsed the Python binary as source and every rank died before the trainer existed. The documented multi-GPU entry point, dead since it shipped.
• DeepSpeed could not train a LoRA model on any stage (#336). With LoRA the no-decay optimizer group is empty; DeepSpeed drops it and torch's scheduler then hits a strict-zip mismatch. Verified repaired on 2×H100 with real soup train: zero2 6790.2 tok/s, zero3 1025.3, zero++ 977.1, all exit 0 with a live adapter (96/96).
• use_fsdp2_compile wrote adapters that reload as all zeros (#335). Under torch.compile the Trainer saves through the wrapper. The tensors were genuinely trained and PEFT matched none of them, emitting only a UserWarning. On 4×H100: 0 of 96 non-zero against 96/96 for the paired non-compile run, reproduced 3/3, the run exiting 0 throughout.
• SGLang returned 500 on every generation (#76). sglang 0.5.16 returns a JSON string; Soup subscripted it as a dict. 100% of requests, and genuinely never run because SGLang does not support Windows.

The vLLM backend now uses your model's chat template (#332, #333)

It hand-rolled a "User: …\nAssistant:" prompt while the transformers backend used apply_chat_template, so every vLLM user's model saw a format it was never trained on. Same server, same sampling params, only the prompt differing: a run-on loop burning all 200 tokens before, an 8-token answer after. Also fixed: finish_reason hardcoded to "stop", --dashboard silently no-opping (/metrics returned 404), and a missing --max-model-len.

New capabilities

• training.seed / training.data_seed (#341) — every run trained at 42 with no knob. Both default to None because an unset seed has to reproduce two different historical defaults.
• Full fine-tuning as lora.r: 0 (#340) — the code path existed but was unreachable.
• --deepspeed zero3_offload — ZeRO-3 with CPU parameter offload. Measured on one H100 (Llama-3.1-8B, bf16, LoRA r=8, 256 steps): 21.65 tok/s at a 38,135 MiB peak.
• trl widened to >=0.14.0,<0.29 (#326), behind a capability-probe compat layer rather than a version table — the version table was wrong twice.

Silent training defects

• data.max_length was capped at 1024 on every SFT run (#78) — max_length is an SFT-only field TrainingArguments does not carry, so it always took SFTConfig's default. Measured: data.max_length=4096 gave 1024 tokens per sample, with no warning.
• training.use_liger: true crashed at step 0 (#78), across the whole supported trl pin — and Liger's architecture match was a substring of the model name, so any locally-loaded model trained without it on a flag the user had set.
• FlashAttention 3 was selected from a version that can never report 3 (#334). On Hopper hardware users silently got FA2 or SDPA while the docs advertised FA3.

Eval, ship and export

• Two of soup ship's three behavioural suites measured nothing (#316) — mini_tool_call and mini_format_json both scored 0.000 on a model that does both correctly. All harness defects.
• The refusal detector missed the apostrophe models actually type (#316). Over the shipped 40-item mini_safety suite, 28 of 40 refusals scored as non-refusals; the suite reported 0.300 for a true refusal rate of 1.000 — a 0.70 error against a 0.05 regression threshold.
• A quantised GGUF export deleted a previously exported f16 (#144). The intermediate was named {model}.f16.gguf — exactly the default output name of a --quant f16 export — and unlinked on completion.

Validated, not changed

• Forward bit-exactness at real model sizes — logits torch.equal against a resident reference of matching numerics at 0.5B, 8B, 14B, 32B and 72B. Every previously published result was on a 3-layer from-config toy. Backward exactness is a separate claim, measured separately — see #331 above and the per-model ledger at the top of the gate record.
• A streamed model is as good as a resident one — paired over five disjoint training subsets and judged by Soup's own soup ship: mean difference +0.006 against an identical 0.013 within-arm spread.
• Against DeepSpeed ZeRO-3 CPU offload, same box, same data, same model: 2.93× the throughput at 9.7× less peak VRAM at matched numerics. Read narrowly — the same session shows eight cards of ZeRO-3 are slower than one card training resident for a model that fits. Layer streaming is not "faster than DeepSpeed"; it is for the case where the one card you have is too small.

Documentation corrected against measurement

• LISA delivers the quality half of its claim, not the memory half (#306). It beats full fine-tuning at both learning rates — and it is 1.22× LoRA's VRAM at 3B and 1.51× at 8B, with the gap widening.
• FlashAttention and Liger were measured for the first time, at 1.015× and 1.051× / −12.9% VRAM, against documented claims of "2–4×" and "20–60% / 20–40%".

Install / Upgrade

pip install -U "soup-cli[train]"

Python 3.10, 3.11 or 3.12. This release adds the upper bound that was missing (#358): requires-python = ">=3.10,<3.13". Those are exactly the versions CI tests. Without a ceiling, pip on 3.13+ resolved PyTorch wheels nobody here has run, and the failure was not a Soup error message — it was a loader crash inside c10.dll / libc10.so before any Soup code executed. If you are on 3.13+, create a 3.12 environment.

Security

No security vulnerabilities were fixed in this release, and none are known outstanding. Two related items:

• Data integrity: the GGUF export defect above (#144) destroyed a user's previously exported f16 file. If you have run soup export --format gguf with a quantised tier next to an existing .f16.gguf, that file is gone; re-export it.
• Supported versions: security fixes now ship for 0.73.x. See SECURITY.md.

Known Limitations

• training.seed reaches the SFT trainer only (#353). Every other task builds its own TrainingArguments; setting it there parses and does nothing.
• A resident 4-bit run is still not bit-reproducible from a seed (#354), while a streamed one is: get_peft_model builds lora_A before Trainer.__init__ calls set_seed. The one-line fix is verified in the gate record but not shipped here. It has a concrete consequence: three runs of one unchanged resident config moved mini_common_sense by 0.375 and mini_mmlu by 0.269 against a forgetting_threshold of 0.05, so five of seven soup ship suites can cross the regression line on a re-run that changed nothing.
• The layer-streaming VRAM pre-flight still over-predicts (#327), and LOGITS_BYTES_PER_ELEMENT was deliberately left at 14 rather than lowered to the measured loss-path value. The asymmetry decides it: over-predicting refuses a config that would have worked — visible, annoying, data intact — while under-predicting on Windows/WDDM raises nothing at all and silently spills to host memory (measured: 9.27 GB allocated on a 4.29 GB card with no exception). Counterfactual: 14 under-predicts 0 of 10 measured rows, the lower value 10 of 10. Practical cost: a streamed DPO run is refused from max_length: 768 on a 4 GB card.
• bitsandbytes still has the defect #331 works around — filed upstream as bitsandbytes#2034.
• DeepSpeed + LoRA is repaired in sft.py only (#336). The other trainer wrappers still hit the empty-parameter-group failure, a user-supplied --deepspeed my.json is passed through unresolved, and ZeRO++ hierarchical partitioning is unit-tested but never exercised across two nodes.
• utils/sglang.py still has both of the defects the vLLM rewrite fixed — its own hand-rolled prompt and a hardcoded finish_reason.
• The closed-loop reward-hacking controller's mechanism is confirmed; its efficacy is not (#286). At 7B the between-mode difference of 0.130 sits inside the within-mode spread of 0.140–0.195.
• Two soup ship leg-2 suites have scoring gaps that outrank capability (#346, #357). mini_tool_call ranks by brace hygiene — Llama-3.1-8B names the right tool 40/40 and scores 0.225 — and mini_mmlu loses 8 of 26 items because extract_mcq_letter does not know \boxed{C}, scoring the 8B at 0.423, below a 0.5B. A third suspected inversion (#356) was withdrawn as a measurement error of our own: it was taken at a 64-token budget where soup ship uses 256.
• The 70B FSDP2 recipe could not be smoke-tested (#41) — it needs all eight cards.
• The RAM-vs-disk streaming throughput gap remains unmeasured (#325), and layer streaming remains BETA.

A note on the preprint

DOI 10.5281/zenodo.21771064 is unaffected in its correctness claims: its configuration is 8B NF4 at 105 MiB per layer, comfortably below the 163.8–171.5 MiB boundary of #331, and it survives a 50-backward soak at worst_abs = 0.0. What this release changes is scope — exactness moves from 3-layer from-config toys to resident references at 8B / 14B / 32B / 72B. A version 2 carrying the H100 validation, the resident references, the DeepSpeed comparison and the disclosed defect is in preparation.

One number should be read with a caveat: the published 119.6 tok/s laptop figure was measured before the #331 repair and has not been re-run on repaired code. The repair cost −4.8% throughput at 32B, so treat it as a pre-repair number until someone re-measures on an RTX 3050. An H100 cannot substitute — the whole point of the H100 result is that this method is transfer-bound, so its throughput does not carry across machines.

---

Full changelog: CHANGELOG.md · Measurement records: benchmarks/