Skip to content

v1.17.0

Choose a tag to compare

View all tags
@albertvillanova albertvillanova released this 29 Oct 15:51
· 12 commits to main since this release
v1.17.0
e257382
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
TRL v1 17 meets Le Chonk

Features

🔥 Mistral Large 4: fine-tune a 1T-parameter model on one node

TRL now supports mistralai/Mistral-Large-4-1T-A52B: 1T parameters, 52B active. The new LoRA example fine-tunes it with SFTTrainer on a single 8×B300 node, on top of its FP8 weights: the experts are split 8 ways with token dispatch, everything else is sharded across the 8 GPUs at load time by transformers' DistributedConfig, and LoRA trains attention and the dense and shared-expert MLPs.

Measured: 32.7k tokens per step in ~8 s, 142 GiB per GPU, with packing and assistant-only loss on reasoning conversations.

mistral-large4-trl_v7_hq.mp4 
training_args = SFTConfig(
    model_init_kwargs={
        "dtype": "auto",  # keep the FP8 weights
        "distributed_config": DistributedConfig(fsdp_size=8, ep_size=8),
    },
    packing=True,
    assistant_only_loss=True,
    ...
)
trainer = SFTTrainer(model="mistralai/Mistral-Large-4-1T-A52B", args=training_args, peft_config=LoraConfig(...), ...)

assistant_only_loss=True works out of the box: TRL ships a training chat template that matches the released checkpoint's, including its reasoning_effort levels (low, medium, high). Needs transformers>=5.22.0, peft>=0.21.0 and accelerate>=1.15.0.

🎯 Unbiased importance sampling under top-k, top-p and min-p

vLLM normalizes its log-probabilities over the tokens left after top_k, top_p and min_p filtering, while the trainer normalized over the whole vocabulary. Every sampled token's log-ratio was off by the log of the probability mass the filter kept, so the vLLM importance sampling ratio was biased towards 0 (around 1e-4 instead of 1 with top_p=0.9). The trainers now normalize over the candidate set each token was sampled from, following Keep Sampling Mask from DeepSeek-V3.2.

  • AsyncGRPO: renormalize log-probs over the top-k/top-p candidate set by @AmineDiro in #7627
  • GRPO: renormalize log-probs over the top-k/top-p candidate set (vLLM colocate mode, vLLM >= 0.28; server mode will follow) by @albertvillanova in #7866
  • Require top_k with min_p in AsyncGRPO so the importance sampling ratio stays unbiased by @albertvillanova in #7937
  • Track the support maximum in the chunked log-prob kernel, so a far-below-max candidate set no longer gives a log-probability of 0 and non-finite gradients, by @kashif in #7983

💥 What this changes for you. vLLM bounds the candidate set it returns with top_k, so with vLLM and the importance sampling correction on (the default), GRPOConfig now raises when top_p < 1.0 or min_p > 0.0 is set without top_k > 0. Set top_k (e.g. top_k=512), or vllm_importance_sampling_correction=False.

⚖️ Score centering is the default AsyncGRPO objective

AsyncGRPO's old log-probs are the vLLM sampler's, so its ratio mixes staleness with the numerical mismatch between the server and the trainer. With an FP8 sampler and a bf16 trainer, the clipped objective collapses: reward goes from 0.60 to 0.34 (FP8 weights) or 0.10 (FP8 weights + KV cache) by step 300, against 0.71 with a bf16 sampler.

Score centering with masked importance sampling is now the default (score_centering=True, importance_sampling_min=0.5, importance_sampling_max=5.0). With it, the FP8 sampler trains exactly like the bf16 one (0.72). It costs 15–20% of generation throughput.

💥 What this changes for you. AsyncGRPO runs now train with a different objective by default, and tokens whose ratio to the sampler falls outside [0.5, 5.0] are dropped. Set score_centering=False to keep the clipped objective.

by @AmineDiro in #7760

🚀 Faster AsyncGRPO rollouts and packing

  • Adaptive in-flight cap from the vLLM load: the rollout worker follows the server's KV cache usage instead of admitting a fixed max_inflight_tasks, so requests stop queueing and being preempted on the server. No new config: everything is read from the server. By @AmineDiro in #7610
  • Filter zero-advantage groups (DAPO's dynamic sampling): groups whose rollouts all got the same reward, or none of which could be scored, no longer cost a forward and backward. By @AmineDiro in #7609
  • Pack the sampling mask ragged and pad it on the device: TreeRow.pack goes from 250 s to 31 s on rank 0 over a 15-step run, and from 1040 ms to 66 ms on a 28k-token multi-turn row. By @AmineDiro in #7798
  • Keep filling a micro-batch when another row has free room by @clementtisseau in #7793
  • Use FlashAttention-4 on Blackwell GPUs in async GRPO and async distillation by @lewtun in #7792

🌳 Tree attention for linear-attention hybrids

Tree packing (v1.16) now supports the Gated DeltaNet hybrids: Qwen3.5, Qwen3.6, Qwen3-Next and OLMo-Hybrid. Since v1.16 the trainer refused them at init; before it, sequence packing accepted them but was silently wrong on them. The causal convolution and the delta-rule recurrence now follow the tree instead of the row, through FLA's chunked kernels.

by @AmineDiro in #7686

🎚️ Precise chunked log-probabilities

The chunked log-probability and KL/JSD heads get the two precision fixes from A Fix for Softmax and Cross-Entropy: confident targets keep their small gradients, and matching teachers produce zero gradients. Entropy-preserving reinforcement learning (Appendix B.4) reports the same high-confidence policy-gradient issue.

  • Fix chunked loss precision by @kashif in #7882
  • Fix the fused log-prob/entropy backward on strided inputs (shifted views from SFT/DPO) by @winglian in #7890

🔀 X-Token cross-tokenizer distillation in GOLD

GOLDTrainer gets off-policy cross-tokenizer distillation based on X-Token: set lmbda=0.0 and xtoken_loss_type="p_kl" or "h_kl" with a precomputed projection matrix.

  • Add X-Token cross-tokenizer KD to GOLD trainer by @kashif in #6126
  • Support byte-fallback tokenizers (Gemma) in GOLD byte-offset alignment by @kashif in #8010

🟥 Training on AMD GPUs

TRL's trainers now run on AMD Instinct GPUs through PyTorch's ROCm build, with a guide and an AMD CI job that runs the test suite on every PR.

🧩 More models

  • Beam: tiny model, SFT/DPO/GRPO/RLOO tests, GRPO tool calling, and a LoRA SFT example on one 8×B300 node. Needs transformers>=5.20.1. By @qgallouedec in #7629
  • Mistral Medium 3.5: training chat template for assistant-only loss by @qgallouedec in #7940
  • LFM2.5-24B-A2B (and LFM2.5-2.6B's current template) for tool calling by @qgallouedec in #7753
  • LFM2.5-1.2B-Instruct and LFM2.5-VL-450M for tool calling by @albertvillanova in #7909, #7910 and #7911
  • TRL's own Qwen3 training template for response parsing, so a tokenizer saved by assistant-only SFT works in GRPO, by @edbeeching in #8007

Other

Removals and requirement bumps

Fixes

Clearer errors

Failure modes made explicit instead of failing silently or deep in the stack, all by @albertvillanova:

  • Raise when LoRA would skip the routed experts of MoE models on transformers v5 with peft<0.19 (#7769), and warn when it skips their dense and shared expert layers with peft 0.19 to 0.21.2 (#7771)
  • Reject a PeftModel passed as ref_model in DPO and KTO (#7808)
  • Reject prompt learning in distillation before wrapping the model (#7718)

Documentation

CI

AMD, all by @albertvillanova: relax the confident-target tolerances for AMD (#7981, #8005)

Minimum versions, all by @albertvillanova: run tests_min_versions on Python 3.11 and pin every dependency at its floor (#7741), including bitsandbytes (#7746), liger-kernel (#7846) and deepspeed (#7855); document it (#7745) and when to add, drop and change the default Python version (#7743)

Docker images, all by @albertvillanova: install Python with uv (#7738), use Python 3.13 (#7739), install kernels through the TRL kernels extra (#7815), remove redundant installs (#7813), skip the uv cache in the release image (#7817), require vLLM >= 0.31.0 (#7820) and explain the floor (#7862), bump uv and upgrade ca-certificates (#7880)

Real objects, not mocks, all by @albertvillanova: real generation outputs in the GOLD and GKD tests (#7902), real config files in the TrlParser tests (#7903), tiny models instead of full-size checkpoints in the GOLD fixtures (#7904), with a new tiny SmolLM3 (#7905, #7906), a real trainer in the IW-OPD tests (#7976), real DistillationLogits in the SDFT tests (#7977), and real objects in the OpenEnv tests (#7985, #7986, #7989, #7990)

MoE LoRA coverage, all by @albertvillanova: test that LoRA on an MoE model trains its routed experts (#7636) and its dense and shared expert layers (#7768), and the error and warning for PEFT versions that skip expert layers (#7860)

Other, all by @albertvillanova: run the test workflows on push to main only (#7848), use the latest Python patch release in pip workflows (#7881), test the math-verify rewards in CI (#7928), check the LFM2.5-2.6B chat template against the Hub (#7907), skip the async GRPO tests on transformers < 5.2.0 (#7750), skip the Liger fixture and tests without an accelerator (#7868), import trl first in the tests so peft 0.17 imports with transformers v5 (#7873), cap the KV cache in the continuous batching tests (#7825), fix a port race in the OpenReward fixture (#7886), pass enable_thinking only to templates that use it (#7898), UTF-8 in tests (#7764), plus GOLD SmolVLM test handling against transformers 5.20.1 (#7728, #7791, #7810, #7929)

New Contributors

Full Changelog: v1.16.0...v1.17.0

Contributors

kashif, winglian, and 9 other contributors
Assets 2
Loading