Skip to content

v1.18.0

Latest
Latest

Choose a tag to compare

View all tags
@AmineDiro AmineDiro released this 13 Nov 15:37
· 11 commits to main since this release
v1.18.0
480dde9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
TRL v1 18 motorcycle release

Features

Warning

The submodules of trl.experimental.async_grpo are gone. from trl.experimental.async_grpo.async_grpo_config import AsyncGRPOConfig, and imports from async_rollout_worker, packing, vllm_client or weight_transfer in that package, no longer work. Import the trainer from trl, the rollout loop from trl.trainer.async_grpo_trainer, and the vLLM client from trl.generation.async_vllm_client (renamed AsyncVLLMClient). from trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer still works, with a FutureWarning, until v2.0.0.

🎓 AsyncGRPOTrainer is now a stable trainer

AsyncGRPOTrainer runs GRPO with generation decoupled from training. A rollout worker in a background process streams completions from a vLLM server, scores them and queues them, while the training loop pulls from the queue and updates the model. Generation and gradient updates overlap instead of alternating, and the trainer sends its new weights to the server over NCCL every weight_sync_steps steps.

In v1.17 it lived in trl.experimental.async_grpo: seven files, three ways to batch and bound generation, two constructor arguments for swapping out internals, and metric names that didn't match GRPO's. This cycle cleaned that up in about 40 PRs, tracked in #7841: config renames, one timeout instead of three, one batching mode, generation held to max_staleness, GRPO's metric names and init checks, a public rollout loop, and the two-file layout every stable trainer has. Then #8288 moved it to the top-level package.

# Before
from trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer

# Now
from trl import AsyncGRPOConfig, AsyncGRPOTrainer

The old path still works and emits a FutureWarning (removal in v2.0.0). The rollout loop extension point lives next to the trainer, from trl.trainer.async_grpo_trainer import AsyncRolloutLoop, Rollout, and isn't exported from trl. The OpenEnv / Harbor harness stays experimental, as HarnessAsyncGRPOTrainer in trl.experimental.async_grpo.openenv_harness. Logged metric names can still change between releases.

Quick start, from the docs. Install with pip install trl[vllm,kernels,math_verify], then start the vLLM server on GPU 0:

CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.5-2B \
    --weight-transfer-config '{"backend": "nccl"}' \
    --logprobs-mode processed_logprobs \
    --api-server-count 8 \
    --max-model-len 4096

Then train on GPU 1 with CUDA_VISIBLE_DEVICES=1 python train_async_grpo.py:

# train_async_grpo.py
from datasets import load_dataset
from trl import AsyncGRPOTrainer
from trl.rewards import accuracy_reward

if __name__ == "__main__":  # the rollout worker is a spawned process that re-imports this script
    dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

    trainer = AsyncGRPOTrainer(
        model="Qwen/Qwen3.5-2B",
        reward_funcs=accuracy_reward,
        train_dataset=dataset,
    )
    trainer.train()

This exact example runs end to end. It's a sanity check, not a speedup comparison:
Model / data / reward Qwen/Qwen3.5-2B, trl-lib/DeepMath-103K, accuracy_reward
Hardware 2x B200: vLLM on one GPU, the trainer on the other
Config default AsyncGRPOConfig: fp32 weights with bf16 autocast, 8 generations per prompt, up to 2048 completion tokens, token_budget read from --max-model-len 4096
Step time ~7 s/step, after ~5 min of warmup
Throughput ~7k trained tokens/s
Trainer vs vLLM log-prob gap 0.007 to 0.02

What the trainer covers, all documented on the trainer page:

  • Tree packing: each micro-batch row is a prefix trie, so a prompt shared by a group's rollouts, or the history repeated by every turn of a multi-turn rollout, is forwarded once. Gated DeltaNet hybrids (Qwen3.5, Qwen3.6, Qwen3-Next) are supported.
  • Agent training: tools, environment_factory and multi-turn rollouts.
  • Score centering with masked importance sampling as the default objective (score_centering=True). score_centering=False gives GRPO's clipped objective with the same masked ratio.
  • Zero-advantage group filtering: groups whose rewards are all equal are dropped before they cost a forward and backward, and their share is logged as frac_zero_std_groups.
  • Top-k / top-p renormalization: with top_p or min_p, set top_k > 0 and the trainer normalizes its log-probs over the candidate set vLLM sampled from.
  • LoRA adapter sync: start vLLM with --enable-lora and each weight sync sends only the adapter (about 160 MB at r=32) instead of merged weights.
  • Custom rollout loops: subclass AsyncRolloutLoop, override filter_prompt, generate_one, generate_one_turn, compute_advantages, setup or close, and set it as loop_cls on a trainer subclass. The trainer's constructor doesn't change.
  • Metrics with reading guides: every logged metric has a one-line definition, plus guides for whether a run is generation-bound or trainer-bound and whether it is saturating or sampling duplicates.
  • Text-only training of VLM checkpoints (Qwen3.5, Qwen3.6, Qwen3-VL), with the vision tower frozen.

💥 What this changes for you. If you used the experimental AsyncGRPOTrainer in v1.17, the breaking changes all landed before the promotion, so they don't have to happen after it. AsyncDistillationTrainer (still experimental) got the same constructor removals, config renames, timeout merge, batching and staleness changes and metric conventions in the same PRs.

Constructor arguments:

  • weight_transfer= is removed. Weight sync is built from the config: NCCL through vLLM's weight-transfer API, or adapter sync for LoRA. The server is a plain vllm serve.
  • rollout_worker= is removed. To change how rollouts are produced, subclass AsyncRolloutLoop and set loop_cls (see above).
  • For OpenEnv / Harbor, HarnessAsyncGRPOTrainer replaces building a HarnessRolloutWorker yourself and passing it as rollout_worker=.

Config:

v1.17 (experimental) v1.18
max_inflight_tasks (-1 = automatic) max_concurrent_rollouts, a fixed ceiling, 512 by default
heartbeat_stale_after_s rollout_worker_timeout (same meaning, same 300 s default)
request_timeout, weight_sync_timeout removed: vllm_server_timeout (default 1800 s, was 240 s) now covers server readiness, every request and every weight-sync call
queue_maxsize removed: generation is held to max_staleness, counted in tokens
fork_threshold_tokens removed: a multi-turn rollout always forks on re-tokenization drift, so no generated token loses its training signal
token_budget <= 0 (fixed-count batching) removed: the token-budget batcher is the only mode. Use None (read max_model_len from the server) or a positive int
repetition_penalty removed: the sampler's log-probs include it and the trainer's don't, which biases the importance ratio
remove_unused_columns=True (ignored) defaults to False, and True raises: every dataset column goes to the reward functions

Behavior to check after upgrading:

  • Readiness used to wait 240 s and a request 600 s. Both now wait up to vllm_server_timeout (1800 s).
  • max_concurrent_rollouts no longer derives from per_device_train_batch_size. With OpenEnv / Harbor, the environment server's MAX_CONCURRENT_ENVS must be at least max_concurrent_rollouts + 1: raise it, or lower max_concurrent_rollouts. The Harbor example's --max-inflight flag is now --max-concurrent-rollouts.
  • rollout/backpressure_s used to measure time waiting on a full queue. It now measures time waiting on the staleness limit.

Metrics: where AsyncGRPO and GRPO log the same quantity, AsyncGRPO now mostly uses the GRPO name. The async-only metrics keep their namespaces (rollout/, sample/, batch/, perf/, tools/, env/), with one meaning per suffix: _count is a count over the logging window, _s a duration, _per_s a rate, _frac a fraction. Dashboards and scripts that read the old names need the new ones.

v1.17 (experimental) v1.18
rewards/<func> rewards/<func>/mean, plus rewards/<func>/group_std
reward_std reward_group_std (within-group, unlike GRPO's batch-wide reward_std)
ratio sampling/importance_sampling_ratio/{mean,min,max}
kl (k3 estimate against the sampler) removed. sampling/sampling_logp_difference/{mean,max} is GRPO's measure of the same gap, and kl stays free for a reference-model KL
frac_reward_zero_std frac_zero_std_groups (GRPO and RLOO keep frac_reward_zero_std)
reward_kept reward_trained
rollout/inflight, rollout/inflight_cap rollout/concurrency, rollout/concurrency_cap
rollout/kv_cache_usage rollout/vllm_kv_cache_usage_frac
sample/rollout_queue_size rollout/queue_size
rollout/generated_tok_s rollout/generated_tokens_per_s
perf/forwarded_tok_s_fwd_bwd, perf/forwarded_tok_s_wall_clock, perf/trained_tok_s_wall_clock perf/fwd_bwd_forwarded_tokens_per_s, perf/wall_clock_forwarded_tokens_per_s, perf/wall_clock_trained_tokens_per_s
perf/mfu_fwd_bwd, perf/mfu_wall_clock perf/fwd_bwd_mfu, perf/wall_clock_mfu
sample/dropped_stale_total, batch/dropped_oversize_total, rollout/vllm_retry_total, tools/<name>/call_total, tools/<name>/failure_total, tools/unknown_name_total the same names ending in _count
rollout/vllm_preemptions_total rollout/vllm_preemption_count
rollout/realign_frac removed, with fork_threshold_tokens

New: num_tokens, completions/{mean,min,max}_length, completions/{mean,min,max}_terminated_length, completions/clipped_ratio, reward_zero_std_groups (the mean reward of the dropped groups) and completions/duplicate_frac. Fixed: tools/call_frequency and tools/failure_frequency now count every rollout, as in GRPO, and rewards/<func>/mean counts once per scored rollout.

The PRs, by @AmineDiro, @qgallouedec and @lewtun unless noted:

  • Constructor and extension point. Drop the weight_transfer protocol (#8065) and AsyncDistillation's rollout_worker argument (#8066), split the rollout loop into generate, reward and advantage stages (#8073), make it public as loop_cls (#8096), drop rollout_worker (#8107), and replace HarnessRolloutWorker with HarnessAsyncGRPOTrainer (#8105), all by @AmineDiro.
  • Config. Rename max_inflight_tasks and heartbeat_stale_after_s and merge the vLLM timeouts (#8063), always fork on re-tokenization drift (#8159), keep only the token-budget batcher (#8168), hold generation to max_staleness in tokens (#8170, #8243), all by @AmineDiro. Drop repetition_penalty (#8139) and default remove_unused_columns to False (#8281) by @qgallouedec.
  • Metrics. GRPO names (#8064), _count and _per_s suffixes (#8186) and namespaces (#8188) by @AmineDiro. Terminated-length metrics (#8247) and per-rollout reward means (#8250) by @qgallouedec. Saturation and duplicate-rollout metrics (#7901) and the zero-std and trained reward renames (#8309) by @lewtun.
  • Layout and promotion. Clean up comments and the accelerator_config code (#8191), inline the rollout worker and packing into the trainer and move the vLLM client and weight transfer to trl.generation (#8274), promote to the stable API (#8288) and announce it in the README (#8292), by @AmineDiro. Rewrite the docs on the layout of the stable trainer pages (#8131), tag the model with trl and async-grpo (#8209), and run the train test against a transformers-backed vLLM server (#8137), by @qgallouedec.

Other

GRPO server mode now renormalizes over vLLM's candidate set. With top_p, min_p or structured outputs, vLLM's log-probs are normalized over the tokens left after filtering. Colocate mode already handled this. In server mode the trainer normalized over the whole vocabulary, so GRPO warned that the importance sampling ratio was biased, or refused structured outputs. A vLLM server (>= 0.28) started with --return-sampling-mask now returns the candidate set of each sampled token, VLLMClient.generate(return_sampling_mask=True) reads it, and GRPO turns it on when the server reports the flag. With Qwen2.5-0.5B-Instruct, top_k=10 and a vLLM 0.32 server, the ratio stays between 0.85 and 1.27 for structured outputs and top_p=0.9, where it was 1e-13 to 1e-31 for structured outputs without the candidate set. trl vllm-serve doesn't add the flag by default, since the server then rejects every request with top_k=0 or temperature=0. By @albertvillanova in #8235

UDPO loss for noisy preferences. DPOTrainer gets loss_type="udpo", the normalized Unbiased DPO loss from Unbiased Alignment for Large Language Models with Noisy Preferences. It corrects for preference labels flipped with probability ε, read from label_smoothing in [0.0, 0.5), and unlike "robust" it stays unbiased when ε is overestimated.

from trl import DPOConfig

training_args = DPOConfig(loss_type="udpo", label_smoothing=0.4)

By @cswjl in #8211

Privileged-context distillation in GOLD. The teacher sees per-example extra context while the student keeps its original prompt, and both score the same completion tokens. Turn it on with use_privileged_context=True, and set privileged_context_column and teacher_prompt_template as needed. It supports text, off-policy (lmbda=0) standard JSD distillation only. See the GOLD docs. By @eshwanthkartitr in #6437

MXFP4 and FP8 checkpoints load dequantized. With kernels installed, any trainer given openai/gpt-oss-20b by name refused to train: transformers rejects full fine-tuning of a quantized model, and PEFT finds nothing to adapt in the packed MXFP4 experts. Trainers now load MXFP4 and FP8 checkpoints dequantized. A quantization_config in model_init_kwargs still wins. FP8 needs transformers>=5.0.0, DeepSeek-V3/R1 still fail to dequantize upstream (huggingface/transformers#49873), and before transformers 5.23 MXFP4 experts come back bfloat16 whatever the dtype. By @qgallouedec in #8070

import trl no longer scans every installed package. It built a package-to-distribution map with importlib.metadata.packages_distributions(), which stats every file of every installed distribution without a top_level.txt (torch 2.14 ships none). The map is gone. Measured on a network filesystem, 8 alternating runs:

before after
import trl 13.1 s 0.08 s
from trl import SFTTrainer 40.7 s 27.4 s

The second row gains less because transformers still makes its own call (huggingface/transformers#49591). By @qgallouedec in #7947

More chat templates for tool calling and assistant-only loss. add_response_schema (so GRPOTrainer with tools) and assistant_only_loss=True in SFT now work on many more checkpoints. All by @albertvillanova:

  • Tool calling: DeepSeek-V3 / DeepSeek-R1 (#7961), GLM-5.3-Flash (#7957), GLM-4.7, GLM-5, GLM-5.1, GLM-5.2 and GLM-5.3 (#7958), Qwen3-Coder (#7964), Qwen3-Thinking-2507 (#7968), Qwen2.5 base (#7972), every Gemma 4 31B, 26B-A4B and 12B revision (#8041), and the Qwen3.6 and Qwen3.8 revisions with generation tags shipped since 2026-11-07 (#8181, #8183)
  • Assistant-only loss: DeepSeek-R1-0528 (#7963), DeepSeek-V3 and DeepSeek-V3-0324 (#7978), Qwen3-Thinking-2507 (#7967), Qwen2.5 base (#7971)
  • Both: the original Qwen3 template, which Qwen3 checkpoints shipped until May 2025 and the base checkpoints still ship (#7974), and Muse Spark Open (#8224, #8225)
  • A tokenizer saved with one of TRL's training templates after SFT is recognized for response parsing, so it works for tool calling and in AsyncGRPO (#8027)
  • get_training_chat_template gains require_prefix_preservation, and SFT uses TRL's stored training template even when the chat template has {% generation %} markers of its own (#8184)

trl <cmd> launches with torchrun. Arguments the script doesn't know go to torchrun (--nproc_per_node, --nnodes, ...), and DeepSpeed and FSDP are set through the training arguments (--deepspeed zero3.json, --fsdp), like for any Trainer. accelerate launch keeps working: until v2.0.0, trl still uses it, with a FutureWarning, for arguments only accelerate knows (--num_processes, --accelerate_config) and when an accelerate config default file exists. examples/accelerate_configs/ is replaced by examples/deepspeed_configs/*.json and examples/fsdp_configs/fsdp2.json. See the CLI docs. By @qgallouedec in #7180

padding_free=True works with max_length in SFT. SFT raised whenever padding_free=True met a max_length (including the default 1024) without packing. Truncation now happens during dataset preparation for padding-free too, so the guard only blocked a valid config. By @qgallouedec in #8077

  • Report Unsloth runs and trainer subclasses under their trainer in telemetry, instead of trl/other, by @qgallouedec in #7926
  • Disable FP fusion explicitly on TRL's Triton kernel launches, so they give the same results on Triton 3.9 and 3.10 (outputs move by at most 3.1e-05, at fp32 rounding level), by @albertvillanova in #8143, #8164 and #8185
  • Use the activation_checkpointing_* methods on transformers 5.23, so GRPO, RLOO, DPO, KTO and Distillation stop emitting a FutureWarning, by @albertvillanova in #8194
  • Only force reentrant checkpointing for PEFT + ZeRO-3 on DeepSpeed < 0.19.3 by @cm8908 in #8120
  • Only call enable_input_require_grads() before transformers 5.0, which handles it, by @qgallouedec in #8252 and @albertvillanova in #8266
  • Load the AsyncGRPO tree attention kernel from flash-attn4 version 1 on Blackwell, which trains Qwen3.8-27B on B200 / B300, by @younesbelkada in #7987

Removals and requirement bumps

vLLM support follows the 5-month window: the minimum goes from 0.22.0 to 0.23.0, and 0.33.x is now supported, so the range is vllm>=0.23.0,<0.34.0. vLLM 0.33.0 requires torch 2.14.0, so pip install -U "trl[vllm]" pulls it.

Fixes

Several of these change what a run trains on. The first one needs action on models you already saved.

MoE models saved by SFTTrainer v1.7.0 to v1.17.1 compute router logits at every forward, inference included. SFT wrote output_router_logits=True and router_aux_loss_coef into the model config and saved them, so fine-tuned MoE models kept returning their router logits, and an overridden coef carried over to the next fine-tune. Like GRPO, RLOO, DPO and KTO, SFT now leaves the config alone. It keeps the config the model was loaded with, though, so a model saved by one of these versions keeps the flag even after another fine-tune. To reset it, set it back to False and save again:

model = AutoModelForCausalLM.from_pretrained("path/to/model")
model.config.output_router_logits = False  # model.config.text_config.output_router_logits for a VLM
model.save_pretrained("path/to/model")

By @qgallouedec in #8103

CISPO gradients were 5x too small by default. CISPO clips the importance weight itself, but an unset epsilon_high fell back to epsilon=0.2, so every on-policy weight was clipped to 0.2. With loss_type="cispo", an unset epsilon_high now defaults to 5.0, the value the docstring already recommended. On the repro, grad_norm goes from 0.0539 to 0.2693 and cispo_clip_ratio from 0.375 to 0. By @qgallouedec in #7917

Assistant-only loss trained tokens the generation prompt already ends with. Several training templates put the turn opener or the pre-filled thinking block inside {% generation %}, so SFT trained tokens the model never generates. The rendered text doesn't change, only the mask: Qwen3 (#8072), GLM-4-MoE (#8122), DeepSeek-R1/V3 and R1-Distill (#8060), Nemotron 3 (#8217), gpt-oss (#8219), Muse Glimmer (#8221), and the Qwen3.6 and Qwen3.8 revisions shipped since 2026-11-07, which bring their own markers (#8199, #8187). The DeepSeek-R1/V3 training template also dropped the reasoning of assistant turns from the target (#8061). All by @albertvillanova.

AsyncGRPO and AsyncDistillation reused prefix-cache KV computed by older weights. Each weight sync pauses vLLM without clearing its prefix cache, so a prefix shared by many prompts (system prompt, few-shot examples, tool schemas) kept the KV of the weights that first served it for the whole run. Rollout requests are now salted by model version. On a Qwen3.5-35B-A3B LoRA run with a ~2.4k-token shared prefix, mean train reward over steps 11-20 went from 0.726 to 0.781. By @AmineDiro in #8300

GKD divided its loss by the length of answers it threw away. On-policy and seq_kd steps counted num_items_in_batch on the dataset labels, then replaced them with generated completions. On the repro, num_items_in_batch goes from 172 to 16, the number of tokens the loss sums over. By @qgallouedec in #7925

One rejected token vetoed a whole VESPO sequence. With loss_type="vespo" and vllm_importance_sampling_mode="token_mask", a token outside the ratio bounds got a ratio of 0, whose clamped log collapsed the sequence weight to ~0, while the token itself kept its gradient. Rejected tokens are now dropped from both the sequence weight and the loss. By @yusufozdil in #8108

GRPO configurations that trained nothing without an error now raise:

  • repetition_penalty with the vLLM importance sampling correction: vLLM applies the penalty to the log-probs it returns, so the ratio dropped to ~1e-28 and the gradient to 0. By @qgallouedec in #7915
  • Structured outputs with the importance sampling correction now need top_k > 0: the ratio fell to about 1e-17. By @albertvillanova in #8145
  • PEFT training separate copies of tied embeddings with vLLM: vLLM >= 0.29 raised during the weight sync, and older versions silently generated from weights the trainer didn't have. By @albertvillanova in #8144

Crashes and hangs:

  • The async trainers hung when the rollout worker died, instead of raising Rollout worker child has failed. By @sergiopaniego in #8038
  • AsyncGRPO and AsyncDistillation crashed at the end of the first step on most MoE families and on Gemma 4, and logged a wrong MFU for hybrids like Qwen3.5. FLOPs are now only estimated for the model types the formula describes, and MFU is omitted otherwise. By @qgallouedec in #8236
  • AsyncGRPO with a model sharded at load time (model_init_kwargs={"distributed_config": DistributedConfig(...)}) failed at init on Qwen3.5/3.6 hybrids, and its merged LoRA weight sync crashed on DTensor base weights. By @AmineDiro in #8294 and #8295
  • AsyncGRPO on Gated DeltaNet models failed on AMD with an AttributeError, by @qgallouedec in #8290
  • The fused LM head and AsyncDistillation raised on MoE configs without num_experts (Granite MoE, granite-4.0-h, PhiMoE), by @qgallouedec in #7919
  • GRPO (beta != 0) and RLOO with a pretrained adapter that has modules_to_save failed at the first vLLM weight sync, by @qgallouedec in #8079
  • DPOTrainer with use_weighting=True and the "sft" loss raised a RuntimeError at the first step, by @Sergasgr in #7651
  • Qwen-VL policies on transformers<5.3.0 could end a completion with <|vision_start|> and crash the forward, by @albertvillanova in #8166
  • The DeepSeek training templates crashed on a tool-call message without a content key, which transformers 5.23 produces, by @albertvillanova in #8228

The rest:

  • GLM-4-MoE responses kept the <|observation|> stop token as content, and the legacy schema (transformers < 5.13) returned tool calls as plain text, by @albertvillanova in #8030 and #8032
  • The experimental SDFT and SDPO trainers logged a biased distillation-loss metric across ranks, by @ucalyptus in #7654
  • The trainers treated peft 0.21.3 as having the MoE dense and shared layers fix, which it doesn't, so they stopped warning while the shared experts went untrained, by @qgallouedec in #8323

Clearer errors

AsyncGRPO checks at init what GRPO checks. All by @qgallouedec: an unsupported train_dataset type (#8204), image datasets, since the trainer trains on text only (#8208), no reward source at all (#8249), GRPO's tool requirements with tools or environment_factory (#8206), a response parser only when there are tools (#8242), the reward count, with a warning when every reward is None (#8205), and a passed PeftModel no longer crashes at the first step while prompt-learning adapters are refused (#8207).

The async trainers raise on a vLLM 4xx right away, with the server's message, instead of retrying a bad request about 90 times and failing with 400, message='Bad Request', by @AmineDiro in #8190. AsyncDistillation fails fast when a teacher has no room to score a sequence, instead of stalling on retried 400 Bad Requests, by @sergiopaniego in #8015.

Documentation

Examples told as stories. Rewrite the gpt-oss example around chess puzzles: on 1,000 test puzzles, gpt-oss-20b goes from 14.8% to 71.8% solved after a full fine-tune, and 69.9% back in MXFP4 (#8085), by @qgallouedec. Add an LFM2.5-8B-A1B research agent example with SFT and GRPO, where held-out exact match goes from 35.8% to 47.8% (#7754), by @qgallouedec. Start the Wordle (#7932) and Sudoku (#8097) examples from SFT models with shaped rewards, add examples/sft_agent_traces to fine-tune a small model on a coding agent's traces (#8222), use the released Open-MOPD experts in the MOPD example (#8014) and move it to its own folder (#8101), by @sergiopaniego. Pin the AsyncGRPO Harbor example to the OpenEnv 0.7.0 release (#8034) and use non-reentrant checkpointing in the Online DPO visual math example (#8268), by @albertvillanova.

CI

Flash Attention compatible tiny models. Make the tiny Llama-3, Llama-3.1, Llama-3.2, Mistral-0.1, Mistral-0.2, Qwen2-R1-Distill, Qwen2SeqCls-2.5, Gemma, Gemma2, Cohere, Olmo3 and SmolLM3 models Flash Attention compatible (#8155, #8156, #8157, #8158, #8160, #8196, #8197, #8198, #8200, #8304, #8305, #8306) by @albertvillanova, and the tiny Qwen2.5, Qwen2.5-VL, Cohere2, GLM-4 MoE, GraniteMoE and PhiMoE ones (#8147, #8148, #8169, #8283, #8284, #8285, #8286) by @qgallouedec. Test SFT on tiny VLMs with images under Flash Attention (#8201) and skip the SFT Flash Attention xfails on torch versions that leak checkpoint hooks (#8303), by @albertvillanova.

Tiny models that look like their references. Add tiny GraniteMoe, PhiMoE and GraniteMoeHybrid models (#7920, #7921, #7922) and quantize the tiny DeepSeek-V3 and GPT-OSS models like their references (#8058), by @qgallouedec. Add a tiny GLM-5.3-Flash script (#8068, #8119, #8125, #8069), compare tiny-model dtypes with references that only ship consolidated safetensors (#8238), write the tiny Mistral Large 4 FP8 checkpoint in the reference layout (#8239), scale the tiny DeepSeek-V3 FP8 blocks to 32x32 (#8240), drop the pinned revisions of the merged tiny models (#8241) and compare saved checkpoints with the reference in check_dtype_pattern (#8118), by @albertvillanova.

Chat template tests. Check the most-downloaded checkpoint of each stored template against the Hub (#7956), derive the training template pairs in the response parsing test (#8084), test that the assistant mask starts where the generation prompt ends, with and without reasoning (#8110, #8124), assert training templates fail like their originals instead of skipping (#8141), and xfail the DeepSeek assistant-mask tests on transformers 5.23.0.dev0 until #8228 (#8215), by @albertvillanova.

Tests that run what users run. Reward with a function, not a reward model, in GRPO and RLOO tests (#8193), train on conversational data in trainer tests (#8195), test Qwen3.5 loaded as a text-only causal LM (#7763), stop the sft_fa2 invariant class at step 4 (#8106) and skip the AsyncGRPO tests when requests is missing (#8291), by @qgallouedec. Build the OpenEnv harness loop with its real constructor (#7988), allow dropped group ids in the OpenEnv session failure test (#8029), fail instead of skip when GOLD test assets don't load (#8090), load the GOLD Countdown examples from Countdown-Task-GOLD (#8091), drop the Llama4 case from the tree packing refusal test (#8138), and skip experimental Qwen3-VL, Gemma4 and LFM2.5-VL tests on old transformers (#8086, #8087, #8088), by @albertvillanova.

Workflows. Install, build and publish with uv (#7883, #7884), fail self-hosted test jobs when pytest doesn't complete (#8006), test vLLM at both ends of the supported range (#7872), run CI when a draft PR is marked ready (#8100), skip the chat templates check on Dependabot PRs (#8152), pin ruff-pre-commit to a commit SHA (#8165) and doc-builder to v0.6.0 (#8154), and narrow then revert the Mistral Large 4 FP8 xfail (#8037, #8092), by @albertvillanova. Bump doc-builder for machine-translated docs (#8013) by @mishig25, and dependency bumps by @dependabot[bot] (#8151, #8153, #8179). Selective activation checkpointing on transformers 5.23 was wrapped and then unwrapped once transformers restored the old behavior (#8094, #8098), by @albertvillanova.

Dead code. Remove is_unsloth_available (#8043), OnlineDPOTrainer.process_vision_row (#8044), IS_SAGEMAKER_MP_POST_1_10 (#8045), the SDPO and SDFT config_cls attributes (#8046), the async rollout workers' _generation_start_time (#8047), WeaveCallback._eval_logger (#8048), an unused unpack in GOLD (#8049) and adjusted_mfu (#8245), by @qgallouedec.

New Contributors

What's Changed

Full Changelog: v1.17.0...v1.18.0

Contributors

albertvillanova, cm8908, and 12 other contributors
Assets 2
Loading