Skip to content

v1.14.0

Latest
Latest

Choose a tag to compare

View all tags
@albertvillanova albertvillanova released this 25 Sep 06:40
· 9 commits to main since this release
v1.14.0
c6a6a16
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

trl.losses is gone: DPO, KTO and GRPO stream their own log-probs

Warning

from trl.losses import FusedLinearDPOLoss (or FusedLinearKTOLoss, FusedLinearGRPOLoss, FusedLinearJSDLoss) no longer works. The module introduced in v1.13 has been removed.

v1.13 vendored Liger's chunked_loss into trl.losses as a holding action, not a destination (#7063). The problem it was holding: those classes reimplement each trainer's loss math, so TRL carried two copies of every formula, and copies drift. That drift is where the bugs were. GRPO's grad_norm differed from the default path, KTO ignored the reference model under PEFT, KTO class weights and DPO label_smoothing were silently dropped. Worst of all, under plain DDP the fused path ran the loss on the unwrapped model, so DistributedDataParallel's reducer was never armed and gradients were never all-reduced: every rank silently kept its own.

The fix is to stop reimplementing. All DPO, KTO and GRPO need from the fused path is per-token log-probs of the selected tokens, and TRL already had _ChunkedLogProbFunction for exactly that: it streams the vocabulary with an online logsumexp and recomputes in the backward pass. Each trainer now runs backbone -> _ChunkedLogProbFunction -> its own existing loss code, unchanged. One implementation of each loss, and the memory win is kept, because full logits are still never materialized.

Thirteen restrictions existed only because the loss had been rewritten in a form that could not express those options. Most are gone: DPO with use_liger_kernel=True now accepts mixed loss types, f-divergences and precomputed reference log-probs, and GRPO gains entropy and off-policy masking. Still refused: use_weighting, compute_metrics, a PEFT adapter on lm_head, prompt-learning PEFT, and the MoE auxiliary loss.

Single H100, Qwen3-0.6B, bs=4, seq 512:

config median step peak
v1.13 fused, inline ref 0.2243 s 7.05 GB
v1.14 chunked, inline ref 0.2457 s (+9.5%) 7.05 GB
v1.14 chunked, precompute_ref_log_probs=True 0.1971 s (-12.1%) 4.83 GB (-31%)

The third row is the point: that configuration did not exist before, because the fused path rejected precomputed reference log-probs outright.

use_liger_kernel=True still works and still enables Liger's model kernels through transformers. In DPO, GRPO and KTO it now selects TRL's chunked log-probability path instead of Liger's fused loss.

  • Stream KTO log probabilities by @kashif in #7075
  • Stream GRPO log probabilities by @kashif in #7077
  • Move DPO to chunked log probabilities by @kashif in #7243
  • Drop the fused Liger JSD path from the experimental distillation trainers by @kashif in #7301
  • Skip frozen chunked log probability gradients by @kashif in #7076
  • Redirect the KTO liger loss through the distributed wrapper under DDP by @qgallouedec in #7247
  • Document what use_liger_kernel actually does in each trainer by @qgallouedec in #7361

A fused Triton logprob + entropy kernel now ships in TRL

The loss head is the only hot path TRL owns: transformers already kernelizes the model internals, and nothing on the Hub covers what happens after the decoder. On one H100 (bf16, V=151936, H=4096, 8192 tokens), selective_log_softmax plus entropy_from_logits cost 12.8 ms and 2.32 GiB over [8, 1024, 151936]. A fused Triton kernel does the same in 0.89 ms, roughly 14x, and lands closer to the fp32 reference than the bf16 path it replaces. GRPO runs this two to four times per step.

It now lives in-tree at trl.kernels and is on by default on CUDA, ROCm and XPU, with the PyTorch implementation as fallback. It was briefly loaded from the Hub; that was reversed because this kernel is pure Triton and compiles at runtime, so the Hub bought nothing while costing a version gate unrelated to whether the kernel works, a download that fails offline, trust_remote_code=True for anyone with kernels installed, a silent except Exception: pass wrapped around all of it, and 13 of 13 tests skipped in CI.

Other

Breaking

Six experimental trainers removed

GRPOWithReplayBufferTrainer, BCOTrainer, PRMTrainer, XPOTrainer, NashMDTrainer and the GSPO-token trainer are gone from trl.experimental.

What changed is not the guarantee, which was always that trl.experimental has none, but the fact that the decision is now written down. Removals used to be argued one pull request at a time, with every thread relitigating the policy before it got to the feature. #7182 settled what the judgment weighs, and #7254 put it in the docs: usage, external issues and pull requests that come from people actually running the feature, whether a stable trainer already covers it, maintenance cost, downstream consumers, owner, and age. Deliberately inputs to a judgment rather than a test: no threshold, no notice cycle, and the numbers stay in the removal pull request where you can check them. A paper index entry outlives the code it described.

The numbers are what make the case. BCOTrainer was 1,600 lines carrying its own reference-model handling, DeepSpeed preparation, tokenization and eval loop, for 11 genuine trainings in two months; it was also the only reason TRL depended on scikit-learn and joblib at all, and both are now gone from pyproject.toml. GRPOWithReplayBufferTrainer had hand-copied _generate_and_score_completions from GRPO and drifted to 385 of 616 lines different, for 8 trainings.

Also breaking, and covered above: trl.losses was removed along the way.

Fixes

  • Gradients were never all-reduced for KTO under plain DDP. compute_loss redirected through the distributed wrapper only for ZeRO-3 and FSDP, so on plain DDP the loss ran on the unwrapped model, DistributedDataParallel.forward() never ran, its reducer was never armed, and every rank silently kept its own gradients. By @qgallouedec in #7247
  • Wrapped packing duplicated data when the input table was sliced. _pack_wrapped sized the output from the sliced view but read from the raw child buffer, so it re-read from the start of the buffer instead of from the slice, and datasets.map(batched=True) passes sliced tables. By @qgallouedec in #6670
  • A reward function logging on only some ranks hung or crashed the run. GRPO and RLOO looped over the keys each rank held and called one collective per key, so ranks called different collectives: log_metric on rank 0 alone hung on NCCL, and log_extra crashed inside gather_object with a 1EB allocation. By @behroozazarkhalili in #7382
  • Special token ids now reach the config the Trainer actually reads, so it no longer realigns them at train time and logs a change the user did not make (part of #7093): eos in the trainers (#7127), written to the text config for composite models (#7315), the reward model pad token (#7318), and the requested pad token applied to vision datasets (#7316, #7352)
  • Do not quantize the teacher by default in the distillation and GOLD entry points by @behroozazarkhalili in #6769
  • Prepare the dataset once per node when the datasets cache is node-local by @qgallouedec in #7241
  • Read the MoE auxiliary loss coefficient from the model config by @qgallouedec in #7248
  • Await async reward functions written as callable classes by @albertvillanova in #7213
  • Fix two stream races in OffloadActivations by @kashif in #7195
  • Only dedupe offloaded activation storages in streams mode by @kashif in #7193
  • Pass the FSDP2 ignored_params kwarg only on torch 2.7 or later by @albertvillanova in #7259
  • Guard the torch 2.6 APIs used by FSDP2 and online DPO by @albertvillanova in #7260
  • Reduce connection-pool warnings in the vLLM client by @vladbataev in #7199

Documentation

  • Document how a feature leaves trl.experimental by @albertvillanova in #7254, the rule the six removals above were made under
  • Document what use_liger_kernel actually does in each trainer by @qgallouedec in #7361
  • Document that AsyncGRPO tools must be module-level functions by @albertvillanova in #7226
  • Remove outdated "under construction" banners from docs by @qgallouedec in #7177
  • docs: fix 'allows to scale' grammar in DeepSpeed guide by @mrchatam in #7196

CI and maintenance

Tiny models, a sweep bringing every tiny test model in line with its reference, all by @albertvillanova:

  • Aligned with their references: Qwen2.5-Coder (#7134), DeepSeek-R1-Distill (#7135), FalconMamba (#7139), Mistral v0.1 (#7203) and v0.2 (#7204), Gemma (#7217), Gemma2 (#7228), OPT (#7233), Phi-3 (#7257), Phi-3.5 (#7276), plus tie_word_embeddings across the board (#7130)
  • Fields mirrored in a second pass: use_mrope (#7332), use_mambapy (#7333), rope_scaling (#7334), cache_implementation and hidden_act (#7335), activation_dropout and prefix (#7336), attention_bias (#7337, #7338)
  • Generators: seeded so regeneration is idempotent and pushes nothing (#7258), pointed at a FalconMamba reference (#7138), head_dim reduced for Gemma (#7216) and Gemma2 (#7227), ffn_dim passed for OPT (#7232), the Phi3 leftover replaced by the 3 and 3.5 variants (#7239), three with no test consumer retired (#7186)
  • Coverage: the canonical GPTNeoX sequence-classification id (#7185), the tiny DeepSeek-R1-Distill trained in the SFT test (#7187), the dtype check skipped for non-safetensors references (#7238), the chat-template tests off tiny Bloom (#7262)

Tests run only when they can be affected, all by @albertvillanova:

  • Skipped when nothing they cover changed (#7171, #7281), and run when test data or shipped package data does (#7283), or when the experimental workflow itself changes (#7278); code quality moved to its own workflow (#7277)
  • Superseded pull request runs cancelled (#7170), the doc build keyed on the pull request number (#7174). The same grouping was applied to main (#7172) and then reverted (#7201), because GitHub buckets cancelled with failure and main looked broken.

Release automation. Publishing now happens on a tag push, and refuses unless VERSION in the tagged commit matches the tag (#7165). Release notes come out sectioned, from a label-to-section map (#7166) with documentation and maintenance labels applied automatically from the changed paths (#7205, #7210). Plus the dev version bump (#7160).

Test reliability, all by @albertvillanova: the rerun filter matches the out-of-memory message rather than the exception type (#7292) and needs pytest-rerunfailures>=16.7 to see wrapped exceptions (#7325); ACCELERATE_MIXED_PRECISION is restored between tests (#7321); the Llava assistant-mask tests are un-xfailed now that transformers fixed them (#7373); non-identity adapters in the use_adapter tests (#7275); MODEL_REVISIONS injected into the PEFT loaders (#7288); the reward model pad token covered (#7350).

Workflow hardening. Token permissions narrowed in the test workflows (#7287), the workflows hardened (#7300), and the upstream issue behind the trufflehog digest pin linked (#7214).

Other. Dependabot bumps (#7192, #7299), the latest-release tests moved from T4 to L40S (#7184) with the Slack action checkout workaround dropped (#7183), the kernel publish job run under a personal namespace (#7343), the token sync comments reworded (#7167) and all three generation-config eos shapes covered (#7168), LM-head gradient buffers and GEMMs skipped when its parameters are frozen (#7076), and the end-of-turn loss mask warning made actionable (#7230).

Dependencies

The supported vLLM window moves from >=0.19.1,<=0.28.0 to >=0.20.0,<=0.30.0.

New Contributors

What's Changed

Full Changelog: v1.13.0...v1.14.0

Contributors

kashif, albertvillanova, and 9 other contributors
Assets 2
Loading