Skip to content

v1.13.0

Latest
Latest

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 10 Sep 00:50
· 8 commits to main since this release
v1.13.0
3d9261f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

Training beyond 1M tokens

image

A new long-context guide plus a runnable example that trains a book-length sequence per step on a single 8×H100 node.

Measured: Qwen3-8B at 1,048,576 tokens, 380 s/step, 56.2 GB per GPU (bf16, per_device_train_batch_size=1, loss_type="chunked_nll"). Runs end to end and saves. The guide documents the levers in the order you hit them (chunked_nll, gradient-checkpointing offload, YaRN RoPE) and the constraints (full attention only, no packing). Needs transformers >= 5.16 for checkpointing offload.

Chunked-CE lm_head projection now runs on tensor cores

The inner loop of the default loss_type="chunked_nll" was doing h.float() @ w.float().t(). Both operands are already bf16, so the upcast bought no information: it moved the GEMM off the tensor cores onto the fp32 SIMT path and materialized an fp32 copy of the whole lm_head weight (2.03 GB for a 248k vocabulary), rebuilt for every chunk and again on every gradient-checkpoint recompute. In an 8×H100 profile of trl sft on Qwen3.6-35B-A3B those two fp32 SIMT GEMMs were 21.6% of all GPU kernel time.

One chunk, 256 tokens × vocab 248,320 × hidden 2048, 1×H100, bf16, fwd+bwd:

fwd+bwd peak mem
before 23.37 ms 5.99 GB
after 3.86 ms (6.0×) 3.03 GB

End to end, 16,384 tokens per step, tokens/s/GPU:

model mode before after
gemma-3-270m (vocab 262k) full FT, 1×H100 24036 31609 1.32×
Qwen3-0.6B (vocab 152k) full FT, 1×H100 21276 26641 1.25×
Qwen3-8B full FT, 2×H100 FSDP2 3554 6009 1.69×
Qwen3-8B LoRA r16, 2×H100 FSDP2 4531 7125 1.57×
Qwen3-30B-A3B (MoE) LoRA attn, 2×H100 FSDP2 4251 5101 1.20×

Same fix applied to the distillation trainers (which paid it twice per chunk, student and teacher). Under accelerate mixed precision the numerics are bit-identical (autocast was already casting the operands back down); without autocast over the projection the GEMM moves to bf16 tensor cores, which is exactly what loss_type="nll" already does.

by @qgallouedec in #6863

Fused linear losses vendored into trl.losses

TRL's only import from liger-kernel was liger_kernel.chunked_loss (the fused linear DPO / KTO / GRPO / JSD losses). That module is pure PyTorch, and most of the GRPO and DPO variants in it were written for TRL. Upstream reviews had stalled, so the code comes home as trl.losses: FusedLinearDPOLoss, FusedLinearKTOLoss, FusedLinearGRPOLoss, FusedLinearJSDLoss (copied from Liger-Kernel v0.8.2, BSD-2 notice kept, bitwise identical to the installed Liger on random inputs).

No new config flag: use_liger_kernel=True keeps selecting the fused loss exactly as before, and still needs liger-kernel installed because transformers' Trainer patches the model kernels with it.

Contributor skills at .agents/skills

Agent skills for contributing to TRL now ship in the repo at .agents/skills.

by @qgallouedec in #6901

Docs: stable vs experimental taxonomy, and how to change the objective

QLoRA on the quantization_config argument

The QLoRA test suites move onto the quantization_config trainer argument added in v1.8, and gain coverage on the VLM paths.

Dependency floors raised

Other

Breaking

⚠️ PPO is removed

PPOTrainer, PPOConfig, and modeling_value_head.py (PreTrainedModelWrapper, AutoModelForCausalLMWithValueHead, AutoModelForSeq2SeqLMWithValueHead) are gone, along with their tests, doc page, and examples.

Bit of a moment for TRL: PPOTrainer predates every PR in this repo. It landed in dfb6a580 on 2020-03-28, the commit that first added the library, back when the package was called lm_ppo. It was the oldest thing in TRL and the last piece of the original codebase.

Why now: unmaintained (no feature work in over a year, every PPO commit since #4482 a drive-by fix), the only trainer never aligned on the input format (it still took tokenized input_ids), near-zero recorded usage, and a magnet for automated bug hunters filing real reports against code nobody runs. Nobody loses anything: from trl import PPOTrainer already stopped working in v1.10, so anyone actually running PPO is pinned to an older TRL and those installs keep resolving exactly as they do today.

create_reference_model stays (BCO, A2PO and Online DPO use it).

by @qgallouedec in #7020

Fixes

  • Fix missing entropy gradient in _ChunkedLogProbFunction backward — backward accepted grad_entropy but never used it, so any loss backpropagating through the entropy output silently got zero gradient from that term, with no error or warning. Only AsyncGRPOTrainer read entropy (under no_grad, for logging), so no user-facing regression today, but this is exported non-experimental code and an entropy-bonus loss term would have hit it silently. By @verma8076 in #6625
  • vLLM weight sync now fails instead of hanging by @AmineDiro in #6913
  • Fail fast on incompatible vLLM for default Async GRPO weight transfer by @DimensionSTP in #6830
  • Fix VLM support in AsyncGRPO by @AmineDiro in #6839
  • Fix variable reference in BCO docs by @albertvillanova in #7046
  • Fix the IW-OPD self-distillation loss assertion in tests by @albertvillanova in #7123
  • Fix typos in help strings and error messages by @qgallouedec in #7146
  • Fix two messages that describe the wrong behaviour by @qgallouedec in #7147
  • Fix three broken doc links by @qgallouedec in #7155
  • Fix the example commands in the script docstrings by @qgallouedec in #7156
  • Correct what the TRL Docker image provides in the Jobs guide by @albertvillanova in #7054
  • Fix paper_index paths and add GOLD, IW-OPD, and AsyncDistillation telemetry by @YeonwooSung in #7051
  • Qualify doc references to experimental classes by @qgallouedec in #7158

Documentation

A docstring and docs consistency pass across the repo, all by @qgallouedec:

  • Fix docstring type specs (#7148), reformat five docstrings to the repo style (#7149), fix docstring content drift (#7150), match config field order to the docstring order (#7151), reflow two docstrings to the doc-builder width (#7153), fix the DistillationTrainer signature annotations (#7152), docs formatting pass (#7154), tidy pyproject / Makefile / .gitignore (#7157)
  • Document the TRL Docker image tags by @albertvillanova in #7055

CI

Docker images — released and dev builds are now separate workflows, the image version comes from the release tag, and the installed TRL version is pinned:

Tiny-model config alignment — a sweep bringing every tiny test model's config in line with its reference model, all by @albertvillanova:

  • Qwen2.5 (#7098), Qwen3 tiny+small (#7106), Qwen3 Instruct-2507 (#7107), GptOss (#7108), GPTNeoX / pythia-14m (#7109), Llama 3.2 + Remote (#7110), Llama 3 (#7112), Llama 3.1 (#7113)
  • Pin layer_types so tiny Gemma3 / Olmo3 cover both attention types (#6962) and tiny Cohere2 (#6963)
  • Show list lengths and mark truncation in the tiny-model config diff (#7105)

Other CI, mostly by @albertvillanova:

  • Use the doc-builder pre-commit hook (#6799), bump its revision (#6969, #7078)
  • Retry tests failing on cuBLAS allocation errors the rerun filter missed (#6916)
  • Use ruff select instead of extend-select to keep the rule set explicit (#6964), stop dependabot from opening ruff pre-commit bumps (#6965), reformat lambda functions for code clarity (#7092)
  • Remove the flash-attn2 xfail markers now that the padding-free tests pass (#7057), fix the invariant suite recording instructions (#7058)
  • Pin the invariant suite's FA2 hub kernel to the v2 branch by @qgallouedec in #6979
  • Filter the benign torch DataParallel scalar-gather warning (#7019)
  • Fix the input metadata of the post-slack composite action (#7029)
  • Group Dependabot GitHub Actions and pre-commit updates in a single PR (#7088), merge the two Dependabot github-actions entries into one (#7091)
  • Dependabot bumps in #6958, #7000, #7079, #7089

New Contributors

What's Changed

Full Changelog: v1.12.0...v1.13.0

Contributors

kashif, albertvillanova, and 8 other contributors
Assets 2
Loading