v0.72.4 — Align on a laptop: DPO, ORPO, SimPO and KTO over layer streaming

What's New

Align a model on a laptop. Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one decoder layer at a time. Until now it ran supervised fine-tuning only; v0.72.4 adds DPO, ORPO, SimPO and KTO.

The whole risk was one thing: DPO needs a reference model, and a second copy of the model would double memory and defeat the entire feature.

• The reference costs nothing. Soup uses the same streamed base with its LoRA adapters switched off — one set of weights, one stream, no second pass. Measured on an RTX 3050 Laptop 4 GB with a 730 MB model:

  	peak VRAM	vs supervised fine-tuning
  streamed SFT	89.53 MB	—
  streamed DPO	81.87 MB	0.914×
  the same run forced to build a real second model	812.32 MB	9.92×

  That third row is the control, and it is what makes the first number mean something: a second instance costs +730.44 MB against 730.44 MB of weights — exactly one copy. The RAM store and the VRAM buffer pool are byte-identical between the SFT and DPO runs.

• KTO is not reference-free, however it is usually described — it selects a reference exactly the way DPO does, so it gets the same treatment and the same memory assertion. ORPO and SimPO genuinely are reference-free.

• Bit-exact against a normal, non-streamed run of the same loss — 0.0 difference for all four. That is the bar every release in this series has to clear.

• The VRAM pre-flight knows a paired loss is twice the rows. DPO, ORPO and SimPO send chosen and rejected through the model as a single tensor, so a budget computed at one row per example would have under-predicted by half — and on Windows the consequence is not an error but a silent spill to host memory that makes the run an order of magnitude slower.

• kto needs batch_size: 2 or more (its KL term is degenerate at 1). Soup says so when your config is read, instead of minutes later after sharding the checkpoint.

• grpo and ppo stay excluded — permanently, not "not yet". Generation rollouts re-read every layer once per generated token, which destroys the amortisation streaming depends on. The refusal says so and deliberately names no release.

The streaming setup now lives in one shared place rather than being copied into each trainer, so the NF4 pre-flight, the RAM/disk tier choice and the VRAM fit refusal cannot drift apart.

Honest cost: the reference is free in memory, not in time. DPO traverses the layer stack three times per step against supervised fine-tuning's two — measured 1.52× the layer reads.

The full measurement record, published as written and including the three measurement attempts that turned out to be invalid, is in benchmarks/gate-v0.72.4-preference-losses.md.

Also fixed — six preference trainers were broken on any recent trl

Found while building this release, and already true before it. Six trainers
(bco, dpo, ipo, kto, orpo, simpo) pass max_prompt_length to their trl
config, and trl removed it in stages — bco at 0.25, kto/orpo/simpo at 0.26,
dpo/ipo at 0.29, which also deleted ORPOConfig and CPOConfig outright:

trl	dpo	kto	orpo	cpo	bco
0.24.0	yes	yes	yes	yes	yes
0.25.1	yes	yes	yes	yes	no
0.26.0	yes	no	no	no	no
0.29.0	no	no	gone	gone	gone

So pip install 'soup-cli[train]' resolving to a recent trl gave you a
soup train --task orpo that failed at import. Nothing caught it because the trl
imports live inside setup(), which no test had ever called on those wrappers — the
end-to-end preference tests added in this release are what exposed it. trl is now
pinned >=0.7.0,<0.25, the last release all six work on. Supporting the newer API is
tracked separately.

Install / Upgrade

pip install -U "soup-cli[train]"

# soup.yaml
base: meta-llama/Llama-3.1-8B-Instruct
task: dpo                     # or orpo / simpo / kto
data:
  train: prefs.jsonl
  max_length: 512
training:
  stream_layers: true
  quantization: 4bit
  batch_size: 1               # kto needs 2 or more
  lora:
    r: 16
    target_modules: [q_proj, v_proj]

soup train --config soup.yaml

Security

No security fixes in this release. The new configuration gates (task allowlist, the KTO batch-size floor) are correctness guards, not security boundaries; the path-containment, symlink-rejection and subprocess handling in the streaming path are unchanged from v0.72.3.

Known Limitations

1. The VRAM pre-flight is a sound upper bound for preference losses, not a tight one. Charging twice the rows at the supervised loss's measured 14 bytes/element is genuinely conservative — the concatenated forward is an SFT-shaped forward at twice the rows — but TRL's preference losses reduce logits to per-token log-probs instead of holding a full-vocabulary fp32 upcast, so their real cost is lower (measured: DPO's whole above-resident cost was 51.76 MB where that charge is ~458 MB for the same shape). On a 4 GB card with a 128k-vocabulary 1B model, DPO at max_length: 512 is allowed (2.63 GB) and from 768 up it is refused (3.60 GB) even though it would probably fit — lower max_length if you hit that. Two attempts to fit a preference-specific constant produced invalid grids (max_length never bound the effective sequence), so no number was published rather than fabricated. Under-predicting is the strictly worse failure, because on Windows it is a silent spill rather than an exception.
2. The reference forward costs time. DPO reads the layer stack 1.52× as often per step as supervised fine-tuning. Streaming makes the reference free in memory only.
3. Measured against trl 0.19.1; CI now resolves 0.24.0 under the new cap. The shipped tests assert the property (no second instance, reference ≠ policy, bit-exactness) rather than TRL internals, and the KTO batch threshold is pinned against whichever trl is installed.
4. The buffer pool is released by Python's cycle collector, not by close() — back-to-back streamed runs in one process (soup sweep, the web UI) hold the previous pool until a collection pass. Bounded by one layer × buffer count, and reclaimed.
5. A full KTO training step over a streamed model is verified on CUDA only. On a CPU-only runner under newer torch/TRL it trips over the streamed base's meta placeholders. Streaming exists to bound VRAM, so a streamed model on CPU is a test convenience rather than a real configuration; KTO's schema gate, setup, reference behaviour and layer-read accounting are all still checked on CPU.
6. Layer streaming remains BETA.