Soup CLI v0.76.1

Soup CLI v0.76.1

A patch release on the v0.76.0 line. It makes the bundled DPO examples train again, puts the same nesting and expansion limits, and a size cap, on every YAML file Soup reads, fixes the declared Typer floor, and loads Hub dataset ids that contain a dot. Everything in v0.76.0 is unchanged.

Highlights

• The bundled DPO examples train again: task: dpo, ipo, bco and preference no longer fail inside trl on examples/data/dpo_sample.jsonl, examples/data/chat_preferences.jsonl or any dataset written the same way (#1849, #1866).
• soup.yaml and every other YAML file Soup reads go through one loader with the same nesting and expansion limits, and a per-reader size cap (#1827).
• The Typer floor is now typer>=0.15.4, and a CI cell starts the CLI on the declared Typer and Click floors (#1802, #1858).
• A single-string data.train that names a Hub dataset with a dot in its name, such as teknium/OpenHermes-2.5, loads from the Hub (#1889).
• The README and the performance docs state the 8B NF4 figure with its SM clock and give the post-#331 row (#1207, #1699).

Fixed

• DPO-family tasks on the bundled examples (#1849, #1866). A string prompt beside chosen / rejected message lists that repeat the user turn failed inside trl after the model had loaded, with an error that changed with PYTHONHASHSEED ('list' object has no attribute 'endswith', or a TypeError). The dpo and kto converters now hand the trainer every preference row in one layout: all strings, or all message lists with the prompt turns in prompt only. reward_model no longer trains on a text with the question in it twice, and orpo and simpo prepare the same tokens as before. A row that cannot be written that way (one answer a string and the other a message list, a malformed message, no prompt to be found) is dropped at load with a reason that names the field, and a dataset that mixes text rows with conversational ones loads as conversations, with a note.
• Hub dataset ids with a dot (#1889). data.train: teknium/OpenHermes-2.5 failed with Data file not found because the loader read the dot as a file extension. A single string now takes the same route as the same name inside a data.train list: a supported file extension (.jsonl, .json, .csv, .parquet, .txt) means a local file, a :// string a remote URI, anything else a Hub id.

Changed

• YAML limits on every reader (#1827). Nesting is limited to 100 levels; expansion to one node per source byte plus 10,000 for anchors, aliases and merge keys, with merge keys chained at most 100 levels. Each reader keeps its own size cap: 1 MiB for the training-config readers and for readers that had no cap before, 256 KiB for recipes, export --quant-config files, RA-DIT configs and sweep files, 5 MiB for agent specs, 16 MiB for can manifests, 64 KiB for README front matter, and 4 MiB for the config read by soup ship. A soup.yaml that is over 1 MiB, nests deeper than 100 levels, cannot be read or is not valid YAML now fails to load with one line naming the file and the problem; the other readers report through their own handlers.
• Typer floor (#1802, #1858). typer>=0.9.0 was declared while no command could start below Typer 0.12.4, and Typer 0.12.4 to 0.15.3 start the CLI only next to Click 8.1 or older. The floor is typer>=0.15.4; click>=8.0.0 was measured again and holds. A fresh install is unchanged. An environment that holds an older Typer now has it upgraded, or the conflict named, at install time, and soup doctor reports such a Typer as outdated.
• README and docs (#1699, #1825, #1826). The 8B NF4 figure is stated with its clock (119.6 tok/s at 952 MHz) beside the post-#331 row (208.6 tok/s at 1935-1957 MHz), with the caveat that the two are not directly comparable. The What's New block of the four README translations describes v0.76.0. Post-release documentation corrections for v0.76.0, no code change.

Security

• Every YAML file Soup reads is now bounded in size, nesting and expansion by one shared loader; a soup.yaml over a limit is refused with one line naming the file and the limit, and the other readers report through their own handlers.
• The audit log finds credential options from the command tree and also masks long options by name, so the values of --slack-url and --discord-url are masked, and --min-tokens, a length threshold, is no longer masked.
• This release is not accompanied by a security advisory.

Known limitations

• A single-string data.train with an unsupported suffix that does not exist (a typo such as data/train.jsnol) now goes to the Hub lookup, as it already did inside a data.train list, instead of failing with Data file not found; offline it reports a connection error. An existing file with an unsupported suffix (data.tsv) still gets Unsupported file format.
• The settings that v0.76.0 accepts with a warning at load (for example lr_groups, early_stop_patience, forgetting_eval_steps) and soup export --format awq / --format gptq are unchanged in this patch: they are still accepted with a warning until v0.77, which refuses or removes them. The v0.76.0 refusals are unchanged too; coming from 0.75.x, run soup doctor --config soup.yaml first.
• A string prompt beside message-list answers is not read when the two answers share leading turns: those turns are the prompt, as orpo and simpo already read such a row, so a question written only in that string has to be in the conversations as well.
• The YAML nesting limit counts the document's top-level mapping as the first level, so 99 brackets under a top-level key is the deepest structure that loads.
• The pixtral-dpo recipe still cannot train (#1867); this patch does not change it.
• docs/data.md does not yet describe the one-layout rule for preference rows.
• How this patch was checked: the DPO fix was run end to end on a tiny random model on CPU, without quantization (dpo, ipo, bco, kto, preference, orpo, simpo and reward_model, two hash seeds), with a control run on v0.76.0 that failed. Not exercised in that check: online_dpo, grpo and ppo (they need a judge or a reward model), runs on a GPU or with 4-bit and 8-bit bases, the oversize-file refusals of the YAML readers other than the training config, and the Typer and Click floor variants beyond Typer 0.15.4 with Click 8.0.0. The test suite was not run in full on the maintainer's machine; the CI run on the release branch covers it.
• No gated work ships in this release, so no record is added under benchmarks/. The preprint (DOI 10.5281/zenodo.21771064) states no measured number that this patch changes and its scope is unchanged; #1699 restates the paper's 8B row with the clock it was taken at, and #1207 tracks the amendment of the preprint as a new version.

Contributors

All seven changes in this release are maintainer pull requests (#1825, #1699, #1826, #1858, #1866, #1827, #1889), plus a correction to three changelog fragments (#1894).

Thank you to:

• @dryn85-max, who reported that every bundled DPO example failed (#1849).
• @umran666, who measured the post-#331 8B NF4 row quoted in #1699 (#1103).

Upgrade

pip install -U soup-cli==0.76.1

A soup.yaml larger than 1 MiB, or nested deeper than 100 levels, now fails to load with one line naming the limit. Coming from 0.75.x, read the v0.76.0 notes first: a config that loaded before can stop loading.

Full changelog: v0.76.0...v0.76.1

v0.76.0 — settings Soup accepted and never applied now stop the load

490 pull requests have merged since 2026-09-12, the day v0.75.0 shipped, and 418 of them came from 80 people outside the maintainer. What they found is the release: settings that Soup validated, documented and accepted, and then read nowhere on the path they claimed to configure, so a soup.yaml could ask for one thing and quietly train another. This release refuses the settings that never took effect, warns about the rest and names the release that will refuse them, evaluates the validation split Soup always held out and never used, and applies the grpo_beta KL penalty to every GRPO variant. It also adds soup adapters check, soup bench train, soup data clean, soup recipes verify and soup rewind, and the layer-streaming disk tier now reads on a background thread, through direct I/O, and can stripe across NVMe drives. A config that loaded on v0.75 can stop loading: run soup doctor --config soup.yaml before you train.

What's New

Breaking changes (read these first)

• A setting that never took effect is refused at load. Each message names the field and what to change. Among them: training.quantization_aware: true (#1222; it never ran quantization-aware training), training.use_longlora (#1240), training.ebft_variant (#1230), training.gdpo_variant (every non-null value), training.rope_scaling_type where no setup reads it (#1697), training.freeze_layers / freeze_ratio outside the text sft / tts path on the transformers backend (#1497, #1581), training.auto_mixed_precision outside sft / tts (#1618), and backend: unsloth on seven further tasks that have no unsloth setup (#1357). Other refusals follow the same rule, for example expand_layers, freeze_trainable_layers, loraplus_lr_ratio and moe_lora combinations that no trainer applies; the changelog lists every one. Settings that are unread but still tolerated for one more release (lr_groups, early_stop_patience, forgetting_eval_steps and 26 other staged fields) warn at load now and are refused from v0.77.
• Validation loss is now evaluated, and grpo, ppo and online_dpo no longer hold rows out. data.val_split held rows out of training on every run, and no trainer on the transformers backend evaluated them (#1223, #1313). Every task that gets a split now evaluates it at the end of each epoch, or every N steps with the new training.eval_steps, and records the val loss; runs take a few percent longer. grpo, ppo and online_dpo evaluate by generating, so by default they neither evaluate nor withhold rows: data.val_split is ignored with a note and every row trains, which changes the rows a default config trains on.
• Every non-standard grpo_variant applies the grpo_beta KL penalty (#1232, #1249). gspo, dapo, dr_grpo, bnpo, two_sided and rft validated grpo_beta and dropped it, so existing variant configs will not reproduce prior runs. grpo_beta: 0 now runs KL-free and skips the reference forward pass (#1247, #1295).
• Exit codes of gate and verdict commands follow one taxonomy: 0 pass, 2 gate failure, regression or drift, 3 usage or input error (#813, #921). soup eval gate and soup eval against fail with 2 (was 1), soup lock check drift is 2 (was 3), soup expect suite failure is 2 (was 3), soup data validate input errors are 3 (was 1), and a mistyped flag on these commands is 3. A CI script that compares the exit code needs a look.
• Other behaviour changes. training.loraplus_lr_ratio now applies on every LoRA task instead of only sft, pretrain and embedding, so those runs produce different weights (#745, #1080). soup export --format torchao and training.nvfp4 resolve their torchao classes from the module that defines them and the [qat] extra now needs torchao>=0.18.0 (#826, #1066). An FP8 request that cannot run, or whose conversion fails, stops the run instead of training in bf16 (#835, #1154; #1152, #1333). soup data preprocess stops on a chat row it cannot tokenize instead of dropping it (#1180, #1182). ReLoRA restarts follow the paper contract and need an explicit quantization: none (#693, #1133). data.mask_history now keeps only the last assistant turn in the loss and requires data.train_on_responses_only: true (#761, #1101). soup data recipe --execute refuses silent placeholder generation (#817, #922). soup bench is now a group: soup bench infer is the old inference benchmark and soup bench <model> still runs it.

New commands and options

• soup adapters check scans a LoRA adapter for silent failure (a near-zero weight delta, dead layers, an all-zero lora_B, orphaned projections) and ends in one contract line, alive or inactive: <reason>, without needing PyTorch (#1721, #1738). soup adapters audit compares what a finished run recorded against the config that started it, and exits 2 on a divergence (#762, #763).
• soup bench train trains a fixed number of steps in a scratch directory and exits 1 when the model was not training (no trainable parameters, a zero or non-finite gradient norm, parameters that did not change). It covers task: sft on the transformers backend (#836, #1146).
• soup data clean sanitizes and repairs a dataset without touching the input file: control characters, zero-width spaces and BOMs by default, and heuristic repairs behind explicit opt-in flags (#718, @Deadpool2000).
• soup recipes verify checks on the meta device, with no weights downloaded, that every shipped recipe, template and example can attach a LoRA adapter, and --config soup.yaml checks one before a long download (#1116, #1117).
• soup rewind names the dataset rows behind a loss spike. An SFT run writes <output>/rewind.jsonl (training.rewind_log, default on) and the command ranks a spike step's rows by their share of its token loss; on an 8 GB M1 it put a poisoned row at 100% of step 58 (#1000, #1018).
• soup infer --batch-size is opt-in batched inference (the default stays 1). soup infer --cuda-graphs and soup bench infer --cuda-graphs are experimental and need PyTorch 2.14 or newer: on one RTX 5070 Laptop (unpinned, Qwen2.5-1.5B in FP16) the slowest run with graphs decoded 2.9x faster than the fastest without; the record is benchmarks/gate-v0.76.0-cuda-graph-decode.md.
• soup ab runs a two-sided, variance-robust test and reaches accept_h0 about twice as soon when there is no difference; --beta is deprecated and ignored, and refused from v0.77 (#1227, #1265, #1418). soup draft measure reports the break-even acceptance rate and the best draft length (#843, #1042).
• Sigstore keyless signing: soup adapters sign --backend sigstore and soup attest emit --sign sigstore, with verification against an out-of-band certificate identity and OIDC issuer (pip install "soup-cli[sigstore]"; #1114, #1115).
• soup train --nodes / --node-rank / --master-addr / --master-port for multi-node Accelerate launches (#40, #853). soup doctor reports GPU architecture and precision capability (#829, #947) and the MLX backend (#659, #660).
• Recipes and training options: 170 ready-made recipes (recipes that could not load their model were fixed or removed, #1398), with kimi-k2.5, deepseek-v4-pro, minimax-m3, mistral-large-3, qwen3.5-35b-a3b and qwen3.6 DPO / GRPO additions; token-chunked checkpointed KL distillation (training.distill_chunk_size, #743); the PEFT LoRA-FA optimizer (#962); an experimental QuEST mixed-route SFT path whose retained result is evaluation-only (#1105); Arabic, Japanese and Simplified Chinese READMEs (#1100, #1733).

Layer streaming (opt-in, still BETA)

The disk tier reads on a background thread, through direct I/O, and can stripe across NVMe drives. On a 36 GB 70B-shaped NF4 store larger than RAM (one RTX 5070 Laptop 8 GB box), a cold step went from 110.5 s on the synchronous read to 16.2-17.3 s, and from 17.4 s to 9.97 s across two drives with SOUP_LAYER_STREAM_STRIPE_DIRS (a sustained-load probe measured 1.28x and 1.41x in two rounds and reached no formal verdict). A store that fits RAM is still faster on the RAM tier, which stream_source: auto picks. The streamed CUDA NF4 forward now follows bitsandbytes' own fused-versus-dequantized dispatch (#842, #1143), and the post-repair 8B NF4 throughput row on a 4 GB card is recorded (#361). Records: direct I/O, striping, sustained probe.

Full list: CHANGELOG.md.

Install/Upgrade

pip install -U soup-cli           # a fresh install with pipx: pipx install soup-cli
pip install -U "soup-cli[train]"  # the training stack

• Run soup doctor --config soup.yaml first. After upgrading and before you train, it loads your config with the new rules and lists every setting that is now refused or warned about, with what to change. A config that must load on both v0.75 and v0.76 needs the refused key removed, not renamed.
• Python 3.10 to 3.12 only (requires-python = ">=3.10,<3.13").
• click is now declared and bounded: click>=8.0.0,<8.5. click 8.5.0 drops the description of every positional argument from every soup <command> --help page, so a fresh install that resolved the newest click showed ADAPTER TEXT [required] with nothing after it (#1790). If another package in your environment needs click 8.5 or newer, install Soup in its own environment (pipx does this).
• typer is declared >=0.9.0,<0.21.0, but only typer 0.15.4 and newer start on every click the metadata admits (#1802). pip install -U soup-cli does not upgrade a dependency that already satisfies its range, so an environment that already holds an older typer fails on every command, soup --help included, with `Type not yet supported: in...

v0.75.2 — outbound endpoint checks refuse private addresses

A security patch release for the 0.75 line. It carries one hardening change and nothing else; everything merged on main since 0.75.0 ships in the next minor release.

What's New

• Outbound endpoint checks refuse private IP literals on every scheme. soup data generate --api-base, the vLLM provider and the commands that use it, judge URLs in soup eval judge, eval-gate suites, soup ship --judge-model / eval.ship.judge_model, training.online_dpo_judge, and the Web UI chat proxy now refuse a private, link-local or reserved IP literal over https as well as http. Loopback still works; address a server on your network by its hostname. training.online_dpo_judge is checked when soup.yaml loads, not only when the trainer starts.
• http://0.0.0.0 is no longer treated as local by soup data generate --api-base and the Web UI chat proxy; use localhost or 127.0.0.1.
• 100.64.0.0/10 and fec0::/10 are non-public for every check that already refused private addresses (webhooks, the OTLP endpoint, telemetry, hub endpoints over HTTP, and soup ingest --pull, where --allow-private-host admits them).
• Non-ASCII spellings of an address are classified as that address. An IP literal written with non-ASCII digits or label separators is read the way the HTTP client folds it to ASCII before it connects. This applies to every check above.

Fixed

• soup ship checks --judge-model before it builds the base and tuned models. It used to load both and only then stop with a usage error.
• The Web UI chat proxy answers 400, not 500, for an endpoint URL that does not parse.

Install / Upgrade

pip install -U "soup-cli==0.75.2"
# or
pipx upgrade soup-cli

Security

• Outbound endpoint validation (GHSA-9f49-hqpf-c849). Upgrade to 0.75.2. Until then, do not run an untrusted soup.yaml, eval-gate suite or .can package that sets judge_model, online_dpo_judge or eval.ship.judge_model, and keep soup ui bound to loopback with its token private.

Known Limitations

• Hostnames are not resolved: a hostname that resolves to a private address is still accepted, as by every other endpoint check in Soup.
• HF_ENDPOINT and the hub endpoint variables refuse private IP literals over plain HTTP only; they come from the operator's environment.
• There is no opt-in for a private TLS endpoint addressed by an IP literal; use a hostname.
• The checks are found by their shape, so a new outbound call could skip them; a ratchet over every outbound call site is #1547.
• Loopback spellings differ between settings: http://[::1] passes the judge and vLLM checks but not eval-gate suites, soup ship or the online-DPO trainer (#1548).
• The refusal names no remedy, and http://0.0.0.0 gets only the generic HTTPS message (#1549).
• IP-literal parsing depends on the platform's inet_aton and, for 6to4 / NAT64 / Teredo addresses, on the interpreter's ipaddress tables (#1550).

Contributors

• @abdugafforov-bobur reported that the outbound endpoint checks accepted an https:// URL naming a private IP literal, with a clear reproduction.

The same fix is on main as #1527.

v0.75.1 — hardening, and cloud runs that keep what they produced

v0.75.1 is a backport, not a cut of main. Its tree is tag v0.75.0 plus 25 cherry-picks, one
test adaptation and the release commits. main was 186 commits past v0.75.0 when this release was
cut, so releasing from main would have shipped an entire minor's worth of unreleased work under a
patch number. The diff of v0.75.0...v0.75.1 is exactly what these notes describe.

What's New

• A layer-streamed LoRA adapter saved as ZERO tensors under peft 0.21.0 (#1005, fixed in #1010).
  peft 0.21 selects an adapter's tensors by the prefixes it reads off model.named_modules(), where
  0.20 filtered by the lora_ substring. The streaming wrapper's module and parameter names carried
  .inner. while its state_dict() keys were canonical, so trainer.save_model(), every
  save_steps checkpoint and get_peft_model_state_dict() returned nothing — and nothing raised.
  Any adapter saved with stream_layers: true on peft>=0.21 before this fix holds zero tensors (a
  40-byte adapter_model.safetensors) and cannot be recovered: re-train it. Naming only; nothing
  on the streamed forward path changes.

• soup runs clean --keep-weights did the opposite of what it said on Click 8.1 (#1057). The
  option was a bare flag with a True default, which Click 8.1 treats as a toggle, so typing
  --keep-weights parsed as False and deleted whole non-best checkpoints. It is now the paired
  --keep-weights/--no-keep-weights: the default and --keep-weights keep weights on every Click
  version, and --no-keep-weights is the one way to delete them. A repo-wide ratchet now fails any
  True-default flag that has no negative form.

• Cloud runs keep their outputs and their cleanup (#1058, with the follow-up #1073 in #1078).
  soup train --cloud modal ran training in an ephemeral container while the local entrypoint
  printed "download checkpoints to <dir>" with nothing to download; outputs now live on the named
  soup-outputs volume, are committed in a finally, and are downloaded with destination
  containment. On Lambda, Ctrl+C during --cloud-submit could kill the lifecycle controller while it
  was inside the finally that terminates the paid instance; the parent now keeps waiting across the
  interrupt, and the follow-up closes the two remaining windows (a signal between Popen returning
  and the guarded region being entered, and one landing on the notice write).

• Hardening across eight areas — see Security below.

Install / Upgrade

pip install --upgrade soup-cli
# or with the training stack
pip install --upgrade "soup-cli[train]"

Security

This release is the fix for security advisory
GHSA-63h4-gvp4-r26g,
which has the full detail, the affected version range of each issue, and upgrade guidance. Summary:

1. Credentials sent to the wrong service. soup push --hub modelscope|modelers resolved the
   Hugging Face token for every hub and presented it to that third party (0.53.10-0.75.0). A judge
   URL of the form https://<host>/<model> was classified as the OpenAI provider for every host,
   so OPENAI_API_KEY was sent as a Bearer token to it (0.33.0-0.75.0). Each hub now authenticates
   only with its own credential, and only api.openai.com receives OPENAI_API_KEY.
2. soup adapters verify --public-key accepted unsigned adapters (0.71.2-0.75.0): the trusted
   key was consulted only inside the ed25519 branch, so a record with any other backend verified
   with exit code 0 even under --strict.
3. The Web UI rendered dataset and run content as HTML (0.7.0-0.75.0) and sent no
   Content-Security-Policy. All rendering now goes through one escaping template, inline handlers
   are gone, and the UI sends a CSP without unsafe-inline for scripts plus X-Content-Type-Options,
   Referrer-Policy and X-Frame-Options; Chart.js is pinned with Subresource Integrity.
4. soup serve tool and adapter routes checked neither Host nor Origin (tool routes
   0.53.7-0.75.0, adapter routes 0.30.0-0.75.0), so a web page the operator visited could reach
   them. They now answer 421 for a foreign Host and 403 for a foreign Origin, and the tool token is
   compared in constant time. Inference routes are deliberately unchanged so a reverse proxy can
   still front them.
   Also (added 2026-09-25): with --tool-auth-token set, GET /v1/adapters,
   POST /v1/adapters/activate/{name} and POST /v1/adapters/deactivate ignored the token in
   0.71.27-0.75.0, so on a non-loopback bind anyone who could reach the port could list adapters and
   switch the one being served. They now require it, so a client that listed adapters without the
   token gets 401.
5. soup mcp serve --allow-execute did not pin every approved input (0.73.3-0.75.0): several
   config paths could change between plan approval and execution, the one-active-execution check read
   only the 50 newest runs, and a bookkeeping failure after spawn left a child running unsupervised.
6. Config regexes could hang loading or training (lr_groups 0.41.0-0.75.0,
   unfrozen_parameters 0.71.23-0.75.0). The old guard was itself the problem — it ran the untrusted
   pattern against a 128-character probe. It is replaced by a structural check that parses the
   pattern and never matches with it.
7. Credential option values were written to the audit log (0.71.3-0.75.0) whenever they did not
   start with hf_, sk- or Bearer. Masking is now by option name.
8. Reading an untrusted .can archive could exhaust memory through YAML alias expansion or
   unbounded member reads, and manifest fields could emit terminal control sequences
   (0.26.0-0.75.0).

Rotate these credentials

• Ran soup push --hub modelscope or --hub modelers on 0.53.10-0.75.0 while logged in to Hugging
  Face or with HF_TOKEN set? Rotate your Hugging Face token at
  https://huggingface.co/settings/tokens.
• Had a judge URL on 0.33.0-0.75.0 pointing anywhere other than api.openai.com while
  OPENAI_API_KEY was set? Rotate that key.
• Passed --tool-auth-token, --auth-token or --api-key on 0.71.3-0.75.0 and collect your audit
  log anywhere? Rotate those tokens and purge the log.

Found by a full-repository review with OpenCodeReview, verified by the maintainer.

Known Limitations

1. 26 tests fail on the maintainer's dev box and none of them is a regression. All are the
   NF4/bitsandbytes/streaming files, and they fail with identical IDs on a clean v0.75.0
   git-archive copy — the control that makes "not a regression" checkable. The underlying divergence
   is #776 (two bitsandbytes 4-bit compute paths disagreeing on sm_120); CI cannot see it because
   those modules are skipif(not torch.cuda.is_available()) and GitHub runners have no GPU.
2. The POSIX SIGINT test for the Lambda controller runs on CI only. It needs a real process group
   and a real signal, so it is skipped on Windows. The platform-independent half (a stderr whose
   first write raises KeyboardInterrupt) runs everywhere.
3. The Web UI token still lives in sessionStorage. The escaping work removes the sinks that
   could read it from the page and the CSP forbids inline script and eval, but the storage choice
   itself is unchanged.
4. Three regex sinks still have no complexity check: eval/custom.py, utils/diagnose/format.py
   and utils/recipe_run.py. The check shipped where config-supplied patterns reach peft and the
   trainers; these three take patterns from other inputs and were deliberately left out of a
   security release rather than swept in.
5. soup can run's extracted config still goes through the generic loader, bounded now by the
   pre-run read_config check, but the loader itself is unchanged.

Release record

• Gate record: no gate. Nothing in this release was decided by a new measurement; every item is a
  correctness or hardening fix with its own test, so no file was added to benchmarks/. Stated here
  rather than skipped silently.
• Preprint: unchanged. No measured number the preprint states moves, and its scope does not
  change — nothing here touches layer streaming, its mechanism or its architecture list.
• Suite: the authoritative signal for the tagged tree is the CI matrix on d1b28dd2, 14/14 green.
  Locally the backport ran 21518 passed / 26 failed on Windows + Python 3.12; the same 26 fail with
  identical IDs on a clean v0.75.0 copy, which is the control that makes "zero regressions"
  checkable. 500 test files, 21734 collected (5 deselected), up from 475 / 21264 at v0.75.0.

Contributors

Every change in this patch release is the maintainer's own work, so there is no contributor list this
time. v0.75.0 — the release this one patches — carried 60 pull requests from 22 people outside the
maintainer, and they are credited in its notes.

v0.75.0 — MLX honours its config; unknown config keys refuse the load

All 60 pull requests in this release came from outside the maintainer, by 22 people. What they found is the release: on the MLX backend, six training options were validated, documented, accepted — and read by nothing, so the same soup.yaml trained a different recipe on Apple Silicon than on a CUDA box, silently. And the deadline v0.74.0 promised is honoured: an unknown config key now refuses the load.

What's New

• Breaking: an unknown config key refuses the load. v0.74.0 reported every key no config model declares — a typo like quantizaton, or a field that only exists on a newer Soup — as a warning that named v0.75 as the release that would start refusing. This is that release: soup train exits 1 before the training stack is imported, and the API / Web UI loader raises ValueError with the same text, naming the field you probably meant. Nothing is defaulted or substituted. The refusal says Refused. rather than Not applied.; the soup sweep guard, which always refused, changes wording the same way. The detector sees a config the way SoupConfig does: a root-level lora: block (the LlamaFactory / Axolotl spelling the schema has accepted and moved under training since v0.40.1) is remapped through the same function before the scan, so a spelling the validator accepts is never one the detector refuses — the release review caught the first draft refusing it. The two soup fetch examples configs that used it (llama-3.1-8b-lora, qwen2.5-7b-dpo) moved to the canonical training.lora; all 167 recipes, 21 templates and examples/configs/ scan clean. Key names are escaped before they reach the terminal (Rich markup and raw ESC bytes in a YAML key could restyle or spoof it), the scan stops at 100 findings and says so, and the Web UI's /api/train/start returns the key and the suggestion instead of a generic error. A config that must stay loadable on v0.74 as well needs the key removed, not renamed. (#627, #879)
• MLX honours the config it accepted. data.train_on_responses_only (#683 by @Srinivasan8888 in #733), warmup_ratio / scheduler / weight_decay / optimizer (#686 in #734), training.max_grad_norm (#749 in #750), gradient_accumulation_steps (#684 by @AmirF194 in #696) and gradient_checkpointing (#685 in #698) were each validated and then dropped on backend: mlx. Only 8 of the 32 optimizer names have an MLX equivalent; the other 24 are refused by name instead of silently becoming AdamW. The response-only mask is Soup's own per-token mask, not mlx-lm's flag — upstream's masks a single prefix and supervises only the last assistant turn on multi-turn chat (measured: 772 supervised tokens unmasked, 71 with upstream's flag, 146 with a correct mask). MLX also drives the live dashboard, the SQLite tracker and the soup ui stream (#23 in #665), and reports validation loss (#739).
• Validation loss existed nowhere. It was computed on every backend and thrown away: on_log read the training loss and never the eval one, and there was no metrics column and no event field to put it in. It is now recorded, streamed and shown on the panel (#23 by @Srinivasan8888 in #713).
• Two guards for the pattern this release is named for. A declared config field that reaches no consumer now fails the suite (#748 by @Srinivasan8888 in #751; #770 records the guard's measured leak, so it is a ratchet, not a proof). And soup doctor --config soup.yaml lists the settings your task/backend does not read, from a declared table — inference over the import graph detected none of five independently-known MLX gaps, which is why the table is written by hand and guarded in both directions (#755 in #756). Scope is deliberately narrow: task=sft on backend=mlx, seven entries.
• Breaking: grpo_variant: gspo is the published sequence-level objective (Qwen Team, arXiv:2507.18071; #723 by @kok-o in #744), replacing a column-centering heuristic in which a padding token also shifted the loss and gradient of every unmasked row sharing its column (#735 by @AmirF194). Existing gspo configs will not reproduce prior runs.
• training.loraplus_lr_ratio crashed every run that set it. It was forwarded to TrainingArguments, which has no such field. It now builds a real PEFT LoRA+ optimizer (#738 by @abdulwaarith0), and optimizer + scheduler state is proven to survive save/resume (#724 by @SID-6921 in #747).
• torch>=2.6.0 closes v0.74.0's headline known limitation: at torch 2.5.1, trl>=0.29 could not import and every preference trainer was dead. CI now proves the floor (#651 by @Samearth17 in #717).
• Data pipeline agreement. soup data validate runs the converters so it agrees with the loader (#712 by @abdulwaarith0) and fails when no row is usable, with an optional minimum-valid-fraction threshold for CI (#811 by @akkupratap323 in #858); a non-dict message drops the row instead of aborting the load, on the multimodal (#670 by @abdulwaarith0) and chatml / audio / video converters (#676 by @BetterAndBetterII in #678); data.interleave no longer places a row on both sides of val_split (#701 by @AmirF194); data.streaming is honoured for a single Hub dataset name (#689 by @jagadeepmamidi in #710); the single-pass validator (#771 by @iam-saiteja).
• Inference prompts are encoded the way training encoded them. soup chat, soup serve and soup infer re-added special tokens to an already-rendered chat template, so tuned models were prompted with a doubled BOS they never saw in training (#781 by @Konuktor in #782).
• ULD cross-tokenizer distillation refuses wasserstein / topk_align on tokenizers that disagree instead of clamping ids into range and training on the teacher's logits for the wrong text with a finite, plausible loss (#704 by @AmirF194), and respects the response-only label mask and causal shift (#711).
• Multipack FFD placement is O(N log N) via a segment tree, with the packing proven unchanged over 6,000 randomized cases — 96x faster at 30,000 rows (#694 by @dchaudhari7177 in #726).
• Also: packing: true on TRL 0.29 (#691 by @jagadeepmamidi in #709); lora.r: 0 full fine-tuning on task: embedding (#700 by @kok-o in #705; #690 by @jagadeepmamidi in #708); PPO forwards training.epochs and ppo_kl_penalty to trl 0.29 (#721 by @be-student in #737); soup eval auto no longer dies with a TypeError after a successful eval (#752 in #754); soup ingest --source langfuse --pull fetches generations live (#204 by @Konuktor in #859); AWQ export requires explicit calibration data (#592 by @Faisal01011); soup doctor reports the [train] extra as one optional group (#854 by @wangzhengzhuo05); terminal plots honour NO_COLOR (#860 by @akkupratap323); soup migrate no longer refuses a valid config for being named *.jsonl (#675 by @swalla02); DeepSpeed's empty-param-group guard receives the model it inspects (#784 by @DYNOSuprovo in #786); the GEMM throughput forecast measures in the card's resolved stream dtype (#617 by @Samearth17 in #648); MLX recipe repo IDs repaired (#661 by @kok-o in #666); soup data validate rejects an unknown --format instead of reporting rows valid for it (#866 by @SID-6921 in #869); four recipes — deepseek-v4-flash-dpo (#662), kimi-k2.6-dpo (#664), qwen3.5-0.8b-grpo and qwen3.5-2b-grpo (#848 by @SID-6921 in #864) — taking the catalog to 167; a Turkish README behind a gate that checks what a translation was synced from (#852 by @Ercaner1988); two repo-wide AST ratchets on path containment (#775 by @Ercaner1988 in #778, #783 by @AaronProbha18 in #789); opt-in hardware-only telemetry, off unless SOUP_TELEMETRY=1 (#318 by @kok-o in #529).

Full list: CHANGELOG.md.

Install / Upgrade

pipx install soup-cli            # or: pip install -U soup-cli
pip install -U "soup-cli[train]" # the training stack: torch>=2.6.0, transformers>=5.16.1, trl>=0.29, peft>=0.20

Python 3.10–3.12. If a config of yours has been printing unknown config key warnings since v0.74.0, fix or remove those keys before upgrading — v0.75.0 refuses them.

Security

• Web UI read endpoints and SSE streams require authentication (#687 by @kok-o in #707). Run configurations, logs and system metrics no longer answer unauthenticated. SSE authenticates via short-lived, single-use tickets exchanged over an authenticated POST, so a durable token never rides in a query string. A non-loopback bind without a valid token is rejected with exit 2.
• soup ui --public no longer serves the FastAPI docs to the LAN (#731 by @Srinivasan8888 in #732). /openapi.json, /docs, /docs/oauth2-redirect and /redoc are absent (404) on a non-loopback bind and unchanged on loopback — removed rather than gated, because /docs is a browser navigation that cannot carry a Bearer header. Reconnaissance, not disclosure: every endpoint the schema described already answered 401 after #707.
• The Web UI training subprocess cannot hang when no client reads its output — stdout is drained by a bounded background buffer (#688 by @kok-o in #706).
• Telemetry is opt-in and hardware-only; SECURITY.md documents the egress policy. Supported version for security fixes: 0.75.x.

Known Limitations

1. The unknown-key refusal is a hard break for a config that was silently carrying an unknown key. v0.74 warned and ignored it; v0.75 refuses it; neither applies it. A config that must load on both needs the key removed.
2. soup doctor --config knows seven settings on one task/backend pair (sft / mlx) and reports nothing elsewhere rather than guessing. The field-consumer guard (#748) has a measured leak (#770): it catches a field nothing reads, not a field something reads wrongly. Its all-clear message overclaims — CUDA-only kernels such as use_liger are unread on MLX and absent from the table (#903).
3. soup plan / soup apply do not run the unknown-key check — they...

v0.74.0 — the base was loaded in fp32 the whole time

116 of the 120 merged pull requests in this release came from outside the maintainer, by 25 people. The maintainer's own four were #484, #486, #498 and #509.

What's New

The frozen base was being loaded in fp32 the whole time (#339 by @blackcoderx in #471). All three from_pretrained sites — text, vision, audio — passed no explicit dtype, so a base that never receives an optimizer step was materialised at twice its checkpoint precision. Measured on an H100 with Llama-3.1-8B + LoRA: 48,241 MiB -> 18,658 MiB peak, 2.59x / 28.9 GB, byte-identical across three repeats. A trainable base still loads fp32, deliberately and documented. The full-fine-tune discriminator is now one shared is_full_finetune() used by both the trainer and the VRAM pre-flight — previously two independent copies that disagreed in both directions.

Transformers 5.x, TRL 0.29, PEFT 0.20 (#502/#503 by @Amix29 in #507). Qwen3.5-family text decoders train on the Transformers path, TRL APIs that moved under trl.experimental are reached by capability probe rather than a version table, and pip install "soup-cli[train,mlx]" resolves again — the two extras previously declared ranges that could not be satisfied together at all.

The free Colab/Kaggle tier could not stream (#425/#429). T4 / P100 / V100 / GTX 16xx crashed stream_layers: true with _amp_foreach_non_finite_check_and_unscale_cuda not implemented for 'BFloat16', because peft creates LoRA adapters in the base checkpoint's dtype while the fp16 GradScaler requires fp32 gradients. Trainable *lora_* params are cast to fp32 before optimizer creation; the full-FT paths are deliberately untouched so trainable memory does not double after the pre-flight.

soup train --cloud lambda (#264 by @kok-o in #528). Plan-only by default. The API key never enters the instance: termination belongs to a local controller, and the finally both terminates and polls to confirm it happened.

Also: soup mcp serve on both mcp majors with the <2 cap lifted, chosen by probing the Server constructor rather than reading mcp.__version__ (#322 in #498) · unknown config keys warned with a "did you mean" suggestion and a stated v0.75 rejection deadline (#627 by @Srinivasan8888 in #628) · per-PR changelog fragments replacing the shared [Unreleased] section (#487 by @Amix29 in #490) · LISA accepting task: pretrain (#307 by @ousamabenyounes in #476) · the DeepSpeed empty-LoRA-group guard extended from one wrapper to all eighteen that needed it (#359 in #484) · fourteen new recipes, catalog now 163.

Install / Upgrade

pip install -U soup-cli
pip install -U "soup-cli[train]"

Security

Four bypasses of the same shape, closed on both validators (#600/#599 by @here-2007 in #604/#624, #616 in #625): abbreviated (127.1), decimal (2130706433), hex (0x7f000001) and octal (0177.0.0.1) IPv4 spellings reached the telemetry and webhook guard — and, through a path the first fix never touched, the OTLP tracing endpoint validator.

Breaking: /v1/tools/bash is re-enabled under real OS-level namespace/sandbox isolation (#151 by @kok-o in #527), and soup serve now exits 2 when bound to a non-loopback host without --tool-auth-token, where it previously only printed a warning. The endpoint it protects now actually executes code, so a warning was no longer a sufficient control.

The SGLang backend also now obeys the --trust-remote-code gate instead of loading every model with it enabled (#360 by @Srinivasan8888 in #619). Breaking by design: a custom-code model on that backend now fails to load without the flag.

Known Limitations

• The declared torch>=2.5.0 floor does not work with trl>=0.29. Measured, not inferred: at torch 2.5.1 torch.distributed.fsdp.FSDPModule does not exist, so trl 0.29.1 cannot import and DPO / KTO / GRPO / BCO are unavailable. pip cannot catch it because trl declares no torch dependency at all, and CI never sees it because >=2.5.0 always resolves to the newest torch. A fresh install is unaffected; a pinned 2.5.x environment is not. Tracked as #651. No floor bump ships here because 2.5.1 was measured to fail and 2.6 was not measured to work.
• soup doctor never reports MLX, and get_mlx_version() always returns "unknown" — #659, fix in review.
• Layer streaming remains BETA.
• The preprint's measured numbers are unchanged, but its scope is now narrower than the code: it states nine streaming architectures and the code admits ten. It stays explicitly scoped to v0.73.0.

Contributors

Every pull request below came from outside the maintainer.

• @Amix29 (34) — the Transformers 5.x / TRL 0.29 / PEFT 0.20 migration and the [train,mlx] resolution conflict; per-PR changelog fragments, which removes the shared-file merge conflict without making changelog loss silent; Apple Silicon monitoring; SmolVLM/Idefics3 vision batches; the two-phase Best-of-N workflow and its durability work.
• @Srinivasan8888 (14) — the unknown-config-key warning with a stated rejection deadline, then returning to cover the one uncovered line of his own guard; the SGLang trust_remote_code gate, which he deliberately kept out of an earlier PR so it could be argued on its own terms; lisa_train_embeddings; four recipes.
• @AmirF194 (13) — live-eval quantization threading; Cut-CE architecture resolution; branch coverage in pure-Python helpers, including one that corrected the issue's own premise.
• @umran666 (7) — reconciling the two reward-hack ladders by establishing what the code actually does rather than adding a mechanism; the log_level foreign-handler branch; the kimi-k2.6-grpo recipe.
• @harshitthek (6) — MLX device detection, where the label was never the harm: every Apple Silicon run silently rewrote quantization: 4bit to none; GEMM repeat selection; path-separator normalisation.
• @ousamabenyounes (4) — LISA extended to continued pre-training; configurable layer-streaming pinning; the physical-RAM ceiling for the RAM tier.
• @blackcoderx (4) — the fp32 upcast, this release's headline; data.interleave wired into training and then extended to streaming and hub-name lists.
• @AchuthReddy-16 (4) — torch_dtype at every production load site plus the CI cell that exercises the transformers floor; mapping-like tokenizer outputs; deferred callback resolution.
• @SID-6921 (4) — the recipe-config snapshot guard and its delta encoding; MLX --resume, which had been silently restarting from scratch.
• @kok-o (3) — the Lambda Cloud backend, whose third revision removed the need for the credential to travel rather than patching the leak; /v1/tools/bash under OS-level isolation.
• @here-2007 (3) — the obfuscated-IPv4 SSRF work, including the tiered guard.
• @Samearth17 (3) — the required PyTorch smoke job, and the published bit-exactness and bitsandbytes reproduction harnesses.
• @ARAVIND281 (3) — the MII backend's chat template and real finish_reason; ORPO/SimPO streaming coverage; the VRAM fit measured on a second GPU and software stack.
• @Faisal01011 (2) — FSDP + QLoRA dtype compatibility; provider sampling in Best-of-N.
• @Nick-800 (2) — eleven recipes across two PRs (Qwen2.5-Coder/Math and the R1-Distill families).
• @CODING-DARSH, @Shutaru, @lesterppo, @Osheun, @taljeon, @k4its1t, @chadetov, @MKnaomi2, @webdevsamran, @YuriPerro (1 each) — cross-tokenizer draft support; Qwen3.5 MoE streaming; the pre-Ampere GradScaler crash; the glm-5.1-dpo recipe; concrete Transformers model-load coverage; extended dtype-guard coverage; the SFT chat-template fixture; a CUDA wheel recommendation the driver can actually run; the peak-GPU figure in the live panel; and the batch probe gated on the measured peak rather than only on OutOfMemoryError.

Full detail, with every entry credited inline: CHANGELOG.md.

v0.73.3 — every pull request came from outside

Every one of the 24 pull requests in this release came from someone other than the maintainer — from eight people, five of whom appear here for the first time. The maintainer's own work in this window arrived as four direct commits (license headers, a test repair, a CI guard), not as pull requests.

What they found is the more interesting number: four separate config flags that were validated by the schema, documented, and then read by nothing.

What's New

Four flags that did nothing

• training.bnb_4bit_use_double_quant was read by nothing. Every 4-bit construction site hardcoded use_double_quant=True, so setting it to false changed your config fingerprint and nothing else. The fix's design choice is the interesting half: the field is Optional[bool] = None, not a plain True, because a True default emits the key into model_dump() and trips the footgun guard on re-validation — measured, 21 of 173 shipped configs stopped round-tripping, which would have broken train --replay and soup sweep. (#321)
• On Apple Silicon, quantization: 4bit was silently rewritten to none. detect_device() did not know MLX, so every run reported "CPU (no GPU detected)" and quietly downgraded. The label was never the harm. The quantization decision is now an explicit, testable resolve_quantization() rather than a branch buried in a 900-line function that has 0% coverage. (#423)
• soup train --no-reexec printed a launch command with your own flags missing. Follow it literally and you trained without --fsdp — and the run succeeded, so nothing pointed back at the hint. Two hand-maintained copies of "what the user typed"; the printed one is deleted and the hint now derives from the argv that actually launches the run. (#372)
• data.interleave validates, documents, parses — and nothing reads it at training time. Found during review of the mix-recipe fix and filed rather than papered over. (#443)

Silent data corruption in assistant-only masking

A tokenizer returning BatchEncoding — which is not a dict — slipped past the type guard, so the label mask was built from the mapping's key strings: no exception, no warning, a normal-looking loss curve, and zero trained tokens. (#430)

Windows: a process that exited with code 259 read as alive forever

259 is also STILL_ACTIVE, so the two were indistinguishable. It defeated run reconciliation and could wedge the MCP execution cap shut with no error an operator could act on. Disambiguated with WaitForSingleObject, and the two ~50-line copies of the liveness check collapsed into one shared module with a test asserting identity, so a third copy fails. (#424)

New: soup mcp serve --allow-execute

Runs a planned training or export behind a short-lived, single-use, server-generated confirmation token bound to the plan and the execution kind — no command, no argv, no shell string, no client-supplied environment. shell=False, stdin=DEVNULL, output to .soup/mcp-runs/<run_id>.log, and the run goes through the existing tracker so soup runs sees what MCP started. (#297)

Also

soup ship's leg-1 noise floor now measured in the judge task modes, not only metric (#403) · eval.ship.noise_floor committable to soup.yaml (#406) · orphaned running MCP runs reconciled on read, because the watcher is a daemon thread killed without unwinding (#401) · the one-active-execution cap gated on a live persisted run, so a restarted server cannot double-book a GPU (#402) · detect_disk_kind seeing through virtio, where a paravirtual device reports rotational: 1 and a measured 1.5 GB/s cloud disk was refused as an HDD (#365) · MitigationLogWriter no longer drops every record when its parent directory vanishes mid-run (#343) · soup draft distill --steps N delivers ~N steps instead of ~N/4.44 (#364) · \boxed {A} with a space before the brace no longer scores as no-answer (#396) · soup env check audits installed versions against Soup's own declared bounds (#368) · bom and attestation as first-class registry artifact kinds so soup card can link them (#309) · two new recipes, qwen3.5-4b-pretrain and deepseek-v4-flash-grpo (#278, #279).

Install / Upgrade

pip install --upgrade "soup-cli[train]"
soup version

Python 3.10–3.12. No config migration is needed and no default changes: bnb_4bit_use_double_quant left unset still resolves to the historical True, so no existing 4-bit run changes behaviour.

Security

• MCP gated execution (#297) closes a real gap between planning and running. The config is snapshotted at plan time and executed from the copy, so the original cannot be edited underneath the run; and digest_file now walks a directory tree by content (sorted relative paths + per-file hash, symlink refusal, bounded) rather than by mtime+size, which did not change when a file inside a protected directory was rewritten — a model could previously be swapped between plan and execution and revalidation still passed. Token consumption and capacity acquisition happen before Popen, so a failed spawn requires a fresh plan rather than enabling replay. --allow-execute is a separate, stronger opt-in than --allow-mutating.
• Windows process liveness (#424) — the 259 collision could wedge the execution cap shut indefinitely. Fixed, and the platform split is deliberate: os.kill(pid, 0) is a genuine existence check on POSIX but on Windows CPython routes signal 0 to GenerateConsoleCtrlEvent, i.e. it sends a console Ctrl+C.

Known Limitations

• #394 (MLX) remains open. The PR in this release hardens the dispatch route so backend: mlx cannot reach the PyTorch/TRL stack; it does not fix the reported hang, and its author retitled it to say so.
• #371 (reward-hack controller) remains open. #414 addresses one fix-path item and is marked Refs, not Fixes.
• #442 — soup data mix --live writes an overlay config that fails to load, the same defect as #330 in a second renderer. Invisible because every --live test mocks subprocess.run, so the artifact is built and never loaded.
• #443 — data.interleave has no training-time reader. This is why the mix recipe collapses to a single dataset rather than emitting the mixture it searched for; it needs a deliberate decision, not another workaround.
• #444 — a CUDA-gated GEMM-ceiling test is flaky under machine load. CI never sees it (it skips without a GPU), which is exactly why it is filed rather than left to surprise someone.

No gate record accompanies this release: nothing here was gated by a new measurement, so benchmarks/ is unchanged. The preprint is likewise untouched — no measured number moves and its scope is unchanged.

Contributors

24 of the 24 pull requests in this release came from outside the maintainer.

Ben Younes (@ousamabenyounes) — eleven merges

• MitigationLogWriter dropped every record in silence once its parent directory vanished mid-run, so the controller kept acting while its evidence stopped growing (#398)
• soup draft distill --steps N delivered only ~N/4.44 optimiser steps — val_split and gradient_accumulation_steps both divide the budget and the epoch arithmetic ignored both (#399)
• The leg-2 MCQ scorer read \boxed {A} as no-answer: LaTeX permits a space before the brace, models emit it, and the cue tier cannot rescue it (#396)
• --noise-floor was the one soup ship gate-policy flag with no config surface, so a team enforcing a floor in CI could not commit it (#410)
• A dead MCP watcher left its run at running forever, because the watcher is a daemon thread killed without unwinding (#407)
• The one-active-execution cap lived in process memory, so a restarted server saw a free slot and could double-book a training already running (#408)
• The leg-1 noise floor was measured in --task-mode metric only, so in the judge modes a win smaller than the instrument's own resolution still counted as a win (#419)
• detect_disk_kind could not see through virtio — a paravirtual device reports rotational: 1 — so a measured 1.5 GB/s cloud disk was classified HDD and refused the streaming disk tier (#411)
• training.bnb_4bit_use_double_quant was validated and then read by nothing: every 4-bit path hardcoded True, so setting it changed the config fingerprint and nothing else (#418)
• soup env check now audits the live environment against the bounds Soup declares about itself, catching pip install vllm quietly pushing transformers past the <5.0.0 cap (#421)
• bom and attestation were not registry artifact kinds, so the two compliance documents soup card exists to publish were the two it could not link (#420)

Darsh (@CODING-DARSH) — eighth merge

• Gated train_execute / export_execute behind a single-use, server-generated confirmation token — no command, no argv, no client environment — with the config snapshotted at plan time and protected directories digested by content rather than by mtime, so a model swapped between plan and execution no longer revalidates (#393)
• Corrected a contributor's handle carried in the previous release's CHANGELOG — a one-line fix to somebody else's credit, which is the kind of thing that no...

v0.73.2 — the release gate stops lying in both directions

soup ship answers one question: did this model get better, or did I break it? Leg 2 — the regression half — was wrong in three separate ways at once. Two of its suites ranked by the wrong thing, one whole failure direction had no detector, and a caller error was indistinguishable from a real regression.

Every item was reproduced against shipped v0.73.1 before a line changed. All four defects live in a scorer, so the faithful instrument is a stub emitting the shapes a real model produced — no GPU needed, and none is claimed.

What's New

Two suites were ranking by the wrong thing

mini_tool_call ranked brace hygiene (#346). Llama-3.1-8B named the right tool 40/40 and scored 0.225. It emitted three opening braces and two closing ones, so the whole-string parse failed, the bounded scan returned the inner object, and the scorer rejected it for having no "function" key. Reproduced here at the extreme: a stub naming the right tool on every item, one brace short, scored 0.000.

The envelope is now restored — but only for an object carrying both name and arguments. Requiring arguments is the safety argument: the prompt shows the model a menu of {"name","description"} objects, so a name-only test would credit echoing the menu back.

mini_mmlu scored an 8B below a 0.5B (#357). 0.423, while the same model scored 1.000 on two other MCQ suites. Two causes: the extractor did not know \boxed{C}, and the prompt never asked for a letter — so of 15 failures, 8 boxed the right letter and 6 boxed a value. A stub answering every item correctly in boxed-letter style scored 0.000.

Both halves are required. The extractor alone is worth +8 items; the prompt alone is worth 0; together 0.423 → 0.731 and the inversion disappears. The new tier fires only on a single A–J letter — reading \boxed{4} as "option 4" would be a wrong credit, not a repair. Among the boxed/cue/paren forms, position decides, not form: a model that boxes a scratch answer and then self-corrects chose the correction.

Leg 2 had no over-refusal detector (#317)

It flagged a drop in refusal rate and had no reverse, so a tune that refuses everything registered as a monotone safety improvement with no ceiling on how useless the model became.

Reproduced as indistinguishability, which is the actual claim: two models with byte-identical scores on all seven shipped suites and the same SHIP verdict — one of which refuses every benign request.

mini_over_refusal is the mirror: 40 hand-authored benign-but-scary-sounding requests (defensive security, first aid, sysadmin verbs like "kill", fiction with dark themes) scored as fraction NOT refused. It joins the default suite (7 → 8), so the existing regression rule catches over-refusal with zero change to the verdict engine. Paired with mini_safety, neither axis can be gamed alone — verified live on a real SmolLM2-135M pair: 1.0000/0.9750 on the benign axis while the harmful axis reads 0.0000 on the same models.

New: soup ship --noise-floor N

Greedy decoding is not deterministic on GPU. Measured on an H100, the same model with no adapter over five runs spread 0.015–0.020 against a threshold of 0.05, with four of six paired deltas sitting inside the spread. The gate was calling differences it could not resolve.

--noise-floor N re-runs the base model N times (2–10), takes each axis's max − min, prints it beside the verdict, and gates every axis at max(threshold, floor). Leg 1 must clear the task axis's floor too.

The max is load-bearing in both directions: a floor above your threshold widens the gate to what is actually measurable, and a floor below it must never tighten the gate behind your back. If a floor exceeds your threshold, the run says so by name.

Scope, stated plainly. The leg-1 floor is measured in --task-mode metric only — in the judge modes a repeat would fold the judge's own sampling noise into a number presented as decode noise. On the dev box the measured floor is 0.0000: CPU greedy decode is deterministic there, and the H100's 0.015/0.020 is not re-claimed as a laptop measurement. A 0.0 floor correctly suppresses nothing.

Caveat carried from the measurement that motivated it: n=3, one model, one dataset. It sizes the effect; it does not calibrate a threshold.

A caller error looked like a finding (#355)

score_bundled_suite returned 0.0 for a non-callable generator on the three behavioural suites, while raising on the MCQ ones. In leg 2 a 0.0 reads as "the model failed every item" → DON'T SHIP — so a caller error was indistinguishable from a regression, and it failed in the direction that looks like a real result. Both branches now raise. A callable that misbehaves is still a failed item, which is the correct existing contract.

Also in this release

• soup data split --stratify-semantic (#388) — partition splits proportionally across semantic groups so a whole topic cannot land in one split. Thanks @Deadpool2000.
• soup mcp serve --allow-execute (#391) — a stronger opt-in than --allow-mutating. Everything stays plan-only in this slice, and the help text says so in the present tense. Thanks @CODING-DARSH.

Two pre-existing bugs found on the way

• The verdict panel never printed its own leg-1 marker. It built the header as ... [{won_str}], and a bare [no win] is valid Rich markup for an unknown tag — so Rich ate it, on every release through v0.73.1. The plain-text rubric, which has no markup parser, printed it correctly the whole time. That asymmetry is why nobody noticed.
• Untrusted names could drive the terminal. rich.markup.escape neutralises [...] and nothing else, so an axis name from an --evidence file carrying raw escape sequences reached the terminal intact.

Install / Upgrade

pip install --upgrade soup-cli

Security

An evidence-supplied noise floor widens the gate, so "floors": {"mini_mmlu": 1.0} in an evidence file would mask any possible drop on that axis. This crosses no new trust boundary — anyone who can edit that file can already forge the raw scores and force a SHIP outright — but it is a far quieter edit to miss in review, and soup ci init wires ship --evidence as a PR merge gate.

Floors are therefore bounded to [0, 1], capped at 50 axes / 256-character names, refused rather than dropped when malformed (a dropped floor replays as a different verdict), and any floor exceeding the threshold is announced by both readers — the CLI on stderr, the MCP tool in its returned payload, since its stdout is the JSON-RPC channel. Neither reader is the quiet one.

Benchmark and axis names are also stripped of control bytes before rendering.

Known Limitations

1. The leg-1 noise floor is measured in --task-mode metric only. In the judge modes the run warns and leaves leg 1 at a 0.0 floor, so a judge-scored win smaller than the judge's own sampling noise is still called a win.
2. n=3, one model, one dataset. The floor sizes the effect; nothing here establishes what N is enough.
3. No GPU floor was measured on the dev box. The run reported is CPU, where greedy decode is deterministic and the floor reads 0.0000.
4. --baseline snapshots taken before v0.73.2 are on a different scale for mini_mmlu, mini_common_sense and mini_tool_call — measured on an unchanged model the jumps are 0.423 → 0.731 and 0.225 → 1.000, far larger than the 0.05 gate. A --baseline supplies the base score from a file and skips the live base run, so it would be diffed against a freshly-scored tuned model. The run now warns by name, but cannot re-measure for you, and there is no version stamp inside a baseline file to detect this automatically. mini_instruction and mini_arithmetic are unaffected (0 of 24 and 0 of 36 items carry a letter answer).
5. An --evidence file remains trusted input. The floor is bounded and announced, but the scores are not signed — --config binds evidence to a config hash, not to the scores.
6. mini_over_refusal is 40 hand-written prompts scored by a keyword refusal heuristic, so it inherits every limit of that heuristic. It detects a collapse in benign helpfulness, not a subtle shift in tone.
7. Two suites still sit at or next to a ceiling after the repair. A suite pinned at 1.000 detects a regression exactly as poorly as one pinned at 0.000, and nothing here re-sizes them.

Measurement record

Published as written, including a withdrawn order-dependence scare, a control that turned out to vary nothing, and a review finding checked against three implementations and partly rejected: benchmarks/gate-v0.73.2-leg2-scoring.md

Tests: 17727 → 17907 (360 files). Full suite green.

v0.73.1 — the free GPU tier, and a formula caught under-predicting

A patch release carrying everything that landed since v0.73.0, headed by a fix that made Soup unusable on the cards most people actually have.

What's New

bf16 was assumed on every CUDA card, in fourteen places (#385/#387). Anything pre-Ampere — T4, P100, V100, GTX 16xx, i.e. the entire free tier on Colab and Kaggle — failed on every task, not just streaming. The trap worth knowing: torch.cuda.is_bf16_supported() defaults to including_emulation=True, so a T4 answers True and the first attempt at this fix was a no-op on the exact hardware it was written for.

New: training.stream_vram_probe (#349) decides the layer-streaming VRAM check by measuring one real forward+backward at your configured shape instead of predicting it. The pre-flight's documented contract is that it never under-predicts. Measured through the real soup train on a 4 GB RTX 3050 (SmolLM2-135M streamed bf16, batch 1):

seq	predicted	real peak	ratio
4352	3.282 GB	3.036 GB	1.081x — over-predicts, safe
5120	3.844 GB	4.118 GB	0.934x — under-predicts
6144	4.590 GB	5.830 GB	0.787x — under by 21%

Under-prediction is the direction that does not announce itself: an OOM on Linux, a silent spill to host memory on Windows/WDDM. The grid guarding that contract could not have caught it — all ten of its rows sit at seq 256 or 512, so it varies batch and says nothing about sequence length.

Opt-in, task: sft only, costs 1.0–5.3 s. Gates on max_memory_allocated, not max_memory_reserved (reserved runs 1.08–1.41x allocated, and gating on it would refuse this feature's own flagship 8B NF4 configuration, which runs). Cannot overrule a prediction more than 4x over budget.

Under use_fsdp2_compile, every checkpoint-* still loaded as a dead adapter (#351). The final save was repaired in v0.73.0; the periodic checkpoints were not, so --resume and load_best_model_at_end silently continued from a re-zeroed lora_B. Measured at 70B: 320 canonical keys in the output root, 320 prefixed ones in checkpoint-100.

The MLX adapter_config.json shipped target_modules unresolved, so a default MLX adapter loaded as a silent no-op (#392). _apply_lora resolved target_modules: auto and trained the resolved modules, while the writer serialised the raw "auto". On load linear_to_lora_layers matches no module against that and load_weights(strict=False) drops every LoRA tensor without a word, so generation with the adapter is bit-identical to the base model. auto is the schema default, so this was every MLX run that did not name its modules by hand. Reported by @armanbot-jpg with a control that isolated it: hand-editing keys in the saved file makes the very same adapters.safetensors produce the tuned behaviour — the weights were fine, the config was not.

backend: mlx never dispatched to the MLX trainer (#362/#363), and training.seed reached the SFT wrapper and nothing else (#353) — so a seeded task: grpo run trained at HF's default of 42, and replicates that differed only in their seed were the same run.

Also: training.batch_size now rejects 0 and negatives (-4 parsed before); Colab's preinstalled torchao breaking peft is mapped to a named fix (#389); the dead calibrated_logits_bytes_per_element hook is wired into the pre-flight (#390/#348); training.stream_vram_override (#386); and three published claims retracted where the evidence did not support them, including "bound by host-to-device transfer". No measured number changed in that retraction.

Install / Upgrade

pip install --upgrade "soup-cli[train]"

Security

• training.batch_size accepted 0 and negative values, which reached the streaming VRAM arithmetic that multiplies by it. Now rejected at config load.
• With the measured probe enabled, a predicted miss no longer refuses before the GPU is touched — so the deferral is capped at 4x over budget, since a shared soup.yaml need not be authored by whoever runs it.

Known Limitations

• stream_vram_probe is task: sft only. The four streaming preference losses keep the fitted prediction — which is exactly where it was measured least accurate at long sequence. Measuring a preference step needs the TRL trainer, which does not exist yet at the point the probe runs. #349 stays open for this reason.
• The mechanism of the long-sequence under-prediction is unidentified. seq**2 from the attention score matrix is the obvious candidate and the numbers do not settle it, so the formula is not fixed — only overrulable. Anything that needs the prediction to be right without a GPU (soup profile, plan-only paths) still gets the old accuracy.
• The probe measures a causal-LM step, not the run's own step — +12.5–14.3% conservative against real SFT training, so it refuses slightly earlier than strictly necessary.
• Windows/WDDM still does not raise on an over-budget allocation, so the refusal direction cannot be OOM-verified on that platform; what is validated is the measured demand.
• Layer streaming remains BETA.

Measurement record

benchmarks/gate-v0.73.1-measured-vram-fit.md — published as written, including the three readings withdrawn during the work (two of which briefly looked like the headline result) and one claim that was wrong in draft and is corrected there.

v0.73.0 — Three days on somebody else's hardware

Every number this project had ever published was measured on one machine: an RTX 3050 Laptop, 4 GB, Windows. From 5–9 August it ran on a borrowed 8×H100 box (Ubuntu 24.04, a much newer torch / bitsandbytes / trl / peft stack) for the first time.

That found one silent correctness defect in layer streaming, four backends that had never actually run, and a documented multi-GPU entry point that had never launched — and it confirmed the headline claim on hardware nothing like the one it was made on.

The full measurement record, published as written including six rejected hypotheses and three false positives that controls caught, is benchmarks/gate-h100-validation.md.

This is a minor bump, not a v0.72.x patch: it adds two capabilities that did not exist and repairs four backends.

What's New

The laptop result reproduces on completely different hardware

Llama-3.1-8B NF4, streamed: 119.6 tok/s in a 3.32 GB peak on the RTX 3050, against a median 113.00 tok/s in the same 3.32 GB on an H100. Layer streaming is bound by host-to-device transfer, not by the GPU — the first evidence of that from outside the original box.

A silent wrong-gradient defect, found and repaired (#331)

On NF4 models above ~165 MiB per layer (32B and up), bitsandbytes.MatMul4Bit kept the packed weight and quant_state on ctx as plain attributes instead of through save_for_backward, so gradient checkpointing could not discard and recompute them. The reference aliased the streaming buffer pool and was read after that slot had been refilled — a bit-exact forward, a healthy-looking loss curve, and wrong gradients on every layer but the last stream_buffers.

Repaired by keeping the weight out of that function entirely (dequantise inside the checkpointed region). Gated against a resident NF4 reference with a repair-disabled control in the same process:

Model	Gradients exact	Control	Peak VRAM	Throughput
Real 32B	256/256	8–12/256	+2.9%	−4.8%
Real 72B	320/320	8/320	+2.6%	−3.7%

Threshold bracketed at 163.8–171.5 MiB/layer; never bf16, never 8B or 14B. De-aliasing was measured and rejected first — bnb holds the reference across the whole forward-to-backward span, so any copy is O(model), taking real 32B from 4,220 to 19,720 MiB.

Four backends that had never actually run, now do

• soup train --gpus N never launched (#77). accelerate launch takes a script path positionally and Soup handed it sys.executable, so accelerate parsed the Python binary as source and every rank died before the trainer existed. The documented multi-GPU entry point, dead since it shipped.
• DeepSpeed could not train a LoRA model on any stage (#336). With LoRA the no-decay optimizer group is empty; DeepSpeed drops it and torch's scheduler then hits a strict-zip mismatch. Verified repaired on 2×H100 with real soup train: zero2 6790.2 tok/s, zero3 1025.3, zero++ 977.1, all exit 0 with a live adapter (96/96).
• use_fsdp2_compile wrote adapters that reload as all zeros (#335). Under torch.compile the Trainer saves through the wrapper. The tensors were genuinely trained and PEFT matched none of them, emitting only a UserWarning. On 4×H100: 0 of 96 non-zero against 96/96 for the paired non-compile run, reproduced 3/3, the run exiting 0 throughout.
• SGLang returned 500 on every generation (#76). sglang 0.5.16 returns a JSON string; Soup subscripted it as a dict. 100% of requests, and genuinely never run because SGLang does not support Windows.

The vLLM backend now uses your model's chat template (#332, #333)

It hand-rolled a "User: …\nAssistant:" prompt while the transformers backend used apply_chat_template, so every vLLM user's model saw a format it was never trained on. Same server, same sampling params, only the prompt differing: a run-on loop burning all 200 tokens before, an 8-token answer after. Also fixed: finish_reason hardcoded to "stop", --dashboard silently no-opping (/metrics returned 404), and a missing --max-model-len.

New capabilities

• training.seed / training.data_seed (#341) — every run trained at 42 with no knob. Both default to None because an unset seed has to reproduce two different historical defaults.
• Full fine-tuning as lora.r: 0 (#340) — the code path existed but was unreachable.
• --deepspeed zero3_offload — ZeRO-3 with CPU parameter offload. Measured on one H100 (Llama-3.1-8B, bf16, LoRA r=8, 256 steps): 21.65 tok/s at a 38,135 MiB peak.
• trl widened to >=0.14.0,<0.29 (#326), behind a capability-probe compat layer rather than a version table — the version table was wrong twice.

Silent training defects

• data.max_length was capped at 1024 on every SFT run (#78) — max_length is an SFT-only field TrainingArguments does not carry, so it always took SFTConfig's default. Measured: data.max_length=4096 gave 1024 tokens per sample, with no warning.
• training.use_liger: true crashed at step 0 (#78), across the whole supported trl pin — and Liger's architecture match was a substring of the model name, so any locally-loaded model trained without it on a flag the user had set.
• FlashAttention 3 was selected from a version that can never report 3 (#334). On Hopper hardware users silently got FA2 or SDPA while the docs advertised FA3.

Eval, ship and export

• Two of soup ship's three behavioural suites measured nothing (#316) — mini_tool_call and mini_format_json both scored 0.000 on a model that does both correctly. All harness defects.
• The refusal detector missed the apostrophe models actually type (#316). Over the shipped 40-item mini_safety suite, 28 of 40 refusals scored as non-refusals; the suite reported 0.300 for a true refusal rate of 1.000 — a 0.70 error against a 0.05 regression threshold.
• A quantised GGUF export deleted a previously exported f16 (#144). The intermediate was named {model}.f16.gguf — exactly the default output name of a --quant f16 export — and unlinked on completion.

Validated, not changed

• Forward bit-exactness at real model sizes — logits torch.equal against a resident reference of matching numerics at 0.5B, 8B, 14B, 32B and 72B. Every previously published result was on a 3-layer from-config toy. Backward exactness is a separate claim, measured separately — see #331 above and the per-model ledger at the top of the gate record.
• A streamed model is as good as a resident one — paired over five disjoint training subsets and judged by Soup's own soup ship: mean difference +0.006 against an identical 0.013 within-arm spread.
• Against DeepSpeed ZeRO-3 CPU offload, same box, same data, same model: 2.93× the throughput at 9.7× less peak VRAM at matched numerics. Read narrowly — the same session shows eight cards of ZeRO-3 are slower than one card training resident for a model that fits. Layer streaming is not "faster than DeepSpeed"; it is for the case where the one card you have is too small.

Documentation corrected against measurement

• LISA delivers the quality half of its claim, not the memory half (#306). It beats full fine-tuning at both learning rates — and it is 1.22× LoRA's VRAM at 3B and 1.51× at 8B, with the gap widening.
• FlashAttention and Liger were measured for the first time, at 1.015× and 1.051× / −12.9% VRAM, against documented claims of "2–4×" and "20–60% / 20–40%".

Install / Upgrade

pip install -U "soup-cli[train]"

Python 3.10, 3.11 or 3.12. This release adds the upper bound that was missing (#358): requires-python = ">=3.10,<3.13". Those are exactly the versions CI tests. Without a ceiling, pip on 3.13+ resolved PyTorch wheels nobody here has run, and the failure was not a Soup error message — it was a loader crash inside c10.dll / libc10.so before any Soup code executed. If you are on 3.13+, create a 3.12 environment.

Security

No security vulnerabilities were fixed in this release, and none are known outstanding. Two related items:

• Data integrity: the GGUF export defect above (#144) destroyed a user's previously exported f16 file. If you have run soup export --format gguf with a quantised tier next to an existing .f16.gguf, that file is gone; re-export it.
• Supported versions: security fixes now ship for 0.73.x. See SECURITY.md.

Known Limitations

• training.seed reaches the SFT trainer only (#353). Every other task builds its own TrainingArguments; setting it there parses and does nothing.
• A resident 4-bit run is still not bit-reproducible from a seed (#354), while a streamed one is: get_peft_model builds lora_A before Trainer.__init__ calls set_seed. The one-line fix is verified in the gate record but not shipped here. It has a concrete consequence: three runs of one unchanged resident config moved mini_common_sense by 0.375 and mini_mmlu by 0.269 against a forgetting_threshold of 0.05, so five of seven soup ship suites can cross the regression line on a re-run that changed nothing.
• The layer-streaming VRAM pre-flight still over-predicts (#327), and LOGITS_BYTES_PER_ELEMENT was deliberately left at 14 rather than lowered to the measured loss-path value. The asymmetry decides it: over-predicting refuses a config that would have worked — visible, annoying, data intact — while under-predicting on Windows/WDDM raises nothing at all and silently spills to host memory (measured: 9.27 GB allocated on a 4.29 GB card with no exception). Counterfactual: 14 under-predicts 0 of 10 measured rows, the lower value 10 of 10. Practical cost: a streamed DPO run is refused from max_length: 768 on a 4 GB card.
• **`bitsan...