v1.16.0

Features

🛠️ TRL now trains on AWS Trainium

The fused LM head from v1.15 is what made Neuron tractable; the remaining gaps are closed here. There is a guide to go with it.

Neuron recompiles once per input shape, so an unpadded run stalls about a minute on every new sequence length. pad_to_multiple_of now defaults to "auto": 128 on Neuron, None elsewhere, so nothing changes on GPU. Trainium2, Qwen3-0.6B SFT, 20 steps: 130 s with the default against 260 s without.

• Add a guide for training on AWS Trainium by @qgallouedec in #7729
• Default pad_to_multiple_of to "auto" by @qgallouedec in #7724
• Run the fused LM head and distillation kernels on devices without Triton by @qgallouedec in #7669
• Report the Neuron device in trl env and telemetry by @qgallouedec in #7672
• Support any accelerator in activation offloading by @qgallouedec in #7670

🌳 Tree attention for AsyncGRPO

AsyncGRPO now packs every micro-batch row as a prefix tree instead of a flat sequence. Multi-turn and multi-generation rollouts repeat a lot of prefix: on a shared prompt, those tokens used to be forwarded once per row. Tree packing forwards them once.

rollout 0:  [ prompt ] [ completion A ]              prompt ──┬── completion A ── later turn
rollout 1:  [ prompt ] [ completion B ]        =>             └── completion B
rollout 2:  [ prompt ] [ completion A ] [ … ]

Chains without a fork collapse into segments, and each segment's queries attend to the K/V gathered along its root-to-segment path in a single FA3 varlen call. No mask tensor: every layer except attention runs once per unique token, and only K/V is duplicated along shared prefixes.

Qwen3-1.7B, 32K tokens per rollout, forward + backward:

layout	packing ratio	sequence packing	tree packing	peak GiB
shared prompt	1.00	8677 ms	8737 ms (0.99×)	35.2 → 35.2
shared prompt	2.29	8266 ms	4341 ms (1.90×)	35.2 → 18.0
shared prompt	3.37	8179 ms	2755 ms (2.97×)	35.2 → 13.6
nested fork	1.60	8557 ms	6217 ms (1.38×)	35.2 → 23.8
nested fork	3.56	8172 ms	2635 ms (3.10×)	35.2 → 13.5

There is no packing option to choose: with nothing shared a tree row is exactly the old sequence packing, at a ~1% cost.

by @AmineDiro in #7328

Muse Spark Open

Support for meta-models/Muse-Spark-Open-816B-A42B: the tiny-model script, coverage in the VLM tests (SFT, DPO, KTO, chat-template utils, fused LM head), and a LoRA SFT example on 8 nodes of 8×H100 with DistributedConfig(fsdp_size=64, ep_size=8), token dispatch and packing. Measured at 14.2 s/step, 131k tokens/step, 40.2 GB per GPU.

Needs transformers>=5.20.0; the adapter save additionally needs transformers#49500 with peft 0.21.2.

by @qgallouedec in #7536

AMD GPUs in the async trainers

• Support AMD GPUs in the async GRPO and async distillation trainers by @Abdennacer-Badaoui in #7689
• Hide the GPU from the async rollout worker on ROCm by @Abdennacer-Badaoui in #7685

Tool-call arguments typed by the tool's JSON schema

parse_response now receives the tools, so tool-call arguments come back typed from the tool's own schema instead of whatever the raw parse produced.

by @qgallouedec in #7650

Other

• Async GRPO tool metrics by @AmineDiro in #7624
• Store async GRPO and distillation rollout sample tokens as numpy arrays by @AmineDiro in #7626
• Save the final step when async trainers stop at num_train_epochs by @lewtun in #7628
• Pass sequence boundaries from the padding-free collator by @qgallouedec in #7599
• Build the reference model with the policy's dtype and attention implementation by @qgallouedec in #7597, and the SDFT/SDPO teacher with the student's by @qgallouedec in #7616
• Find unused parameters under DDP for vision-language models by @qgallouedec in #7598
• Pad the generation prompts to pad_to_multiple_of by @qgallouedec in #7671
• Drop the client-side observe loop from the CARLA examples by @sergiopaniego in #7721
• Use None for unscored env.reward in OpenReward by @albertvillanova in #7541

Removals and requirement bumps

• Remove the experimental SSD trainer by @qgallouedec in #7706
• Refuse nn.DataParallel in RewardTrainer by @albertvillanova in #7571
• Drop vLLM 0.21.0 support by @qgallouedec in #7700
• Require accelerate>=1.10.1 and drop the ParallelismConfig shim by @albertvillanova in #7643
• Declare a minimum bitsandbytes version by @albertvillanova in #7641
• Declare the direct dependencies TRL gets transitively by @albertvillanova in #7713

Fixes

• Fix silently wrong torch.compile results in the Triton loss kernels on torch < 2.11. torch_compile=True trained with loss=0.0 and grad_norm=nan at every step. Inductor passes scalar kernel args as constexpr before torch 2.11, the .to(tl.float32) casts raised on those, and Inductor then assumed every input was mutated and miscompiled. By @albertvillanova in #7688
• Fix FSDP2 vLLM colocate weight sync for tied-embedding models by @Abdennacer-Badaoui in #7647
• Prevent VLM policies from generating the image placeholder token in GRPO, RLOO and Distillation by @albertvillanova in #7695
• Fix Online DPO VLM training when the prompt exceeds max_length (#7703) and for processors with 5D pixel values (#7704), both by @albertvillanova
• Fix activation offloading on torch < 2.4 by @albertvillanova in #7748
• Fix torch.compile(fullgraph=True) of selective_log_softmax_and_entropy on torch 2.6 by @albertvillanova in #7719
• Keep the Triton loss kernels in fp32 under torch.compile by @qgallouedec in #7600
• Read is_triton_available once in the fused LM head kernels so Dynamo does not warn by @qgallouedec in #7731
• Fix Distillation for models split across devices by @albertvillanova in #7573
• Restore the gradient checkpointing kwargs after generation by @qgallouedec in #7668
• Drop the FlashAttention warnings for padding-free and packing in SFT by @qgallouedec in #7620
• Restore the distributed env vars that vLLM colocate tests leave behind by @Abdennacer-Badaoui in #7723

Clearer errors

A sweep making failure modes explicit instead of crashing deep in the stack, all by @albertvillanova:

• Raise a clear error when the fused LM head is used without Triton (#7639)
• Raise a clear error for prefix tuning on peft versions that cannot support it (#7646)
• Raise a clear error when the Distillation student and teacher heads are on different devices (#7605)
• Reject a text-only teacher in GOLD VLM distillation instead of crashing (#7581)
• Name the missing vLLM server setting when the server cannot serve TRL (#7611)
• Raise early for use_kernels with transformers 5.2.0 and 5.3.0 (#7577), and if kernels is installed with transformers 5.3.0 in the tiny NemotronH scripts (#7575)

Documentation

• Add a guide for training on AWS Trainium by @qgallouedec in #7729
• Add guidance on testing through the public API to the contributing guide and agent instructions by @albertvillanova in #7606
• Document the criterion for raising a dependency version floor by @albertvillanova in #7638

CI

Real objects, not mocks. The bulk of this cycle, applying the repo's testing rule across the suite, all by @albertvillanova: run the GRPO entropy mask tests on a real trainer instead of a Mock (#7566), the teacher logprob tests on a real trainer instead of a MagicMock (#7579), build real GOLD trainers in the init (#7582), input preparation (#7614), on-policy generation (#7613, #7615), output decoding (#7633), ULD helper (#7634) and no-backend tokenizer (#7635) tests, check GOLD dataset preparation through the trainer (#7632), and build real async trainers in the rollout state checkpoint tests (#7583)

Coverage, all by @albertvillanova: test SFT (#7552) and DPO/KTO/Reward/GRPO/RLOO (#7601) on a model split across devices, test fp16 training across six trainers (#7572)

Experimental suite on main, all by @albertvillanova: run the experimental tests on pushes to main (#7675), nightly (#7681), on demand (#7678) and when their Makefile target or dependencies change (#7680), and report failures to Slack (#7676)

Other: use Python 3.13 by default in CI by @qgallouedec in #7648, remove the slow test infrastructure (#7604), rename the distributed smoke tests to distributed tests (#7631), skip the kernel tests when Triton is not installed (#7563), guard the aiohttp import in the experimental async rollout workers (#7578), drop use_cpu and bf16 settings the CPU fixture already covers (#7612), re-record the sft_fa2 invariant reference after the fused LM head by @qgallouedec (#7591), plus warning filters in #7586, #7588, #7590, #7645, #7727

New Contributors

• @Abdennacer-Badaoui made their first contribution in #7647

Full Changelog: v1.15.0...v1.16.0