Skip to content

v1.11.0

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 26 Aug 20:02
· 6 commits to main since this release
v1.11.0
d0a2cd5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

Use vLLM's own server instead of TRL's

trl vllm-serve used to run a custom FastAPI app around vLLM's LLM, with its own data-parallel fan-out and a worker extension for weight sync. vLLM's own server covers all of that. The custom server is gone: trl/scripts/vllm_serve.py drops from 1218 to ~130 lines and now translates its flags into vllm serve, prints the equivalent command, forwards any extra argument, and sets the three settings TRL needs (--weight-transfer-config, --logprobs-mode processed_logprobs, --max-logprobs -1). fastapi, uvicorn and pydantic leave the vllm extra.

Weight sync now uses vLLM's NCCL weight-transfer engine — tensors are announced once and streamed in a single packed broadcast, instead of one HTTP request + broadcast per tensor. Distillation teacher logprobs come from prompt_logprobs.

Benchmark (GRPO server-mode on H100, Qwen2.5-1.5B / Qwen2.5-VL-3B, same seed, same data):

Setup before after speedup
Text, TP=1 1.08, 0.91 s/step 0.63, 0.62 s/step 1.59×
Text, TP=2 0.96, 0.96 s/step 0.62, 0.60 s/step 1.57×
VLM, TP=1 2.14, 2.13 s/step 1.48, 1.48 s/step 1.44×

Bit-identical completions vs the old server (16/16 over 4 prompts × n=4, logprobs identical to 0.0 over 499 tokens). trl vllm-serve still accepts the same flags — it just forwards to vllm serve now.

by @qgallouedec in #6765

New experimental AsyncDistillationTrainer (+ multi-teacher MOPD)

Async on-policy distillation, architected like AsyncGRPOTrainer: a background rollout worker generates the student's own completions and scores them against a teacher served over HTTP, so generation and training overlap instead of alternating. The teacher is never loaded locally — just a vLLM server URL.

Also supports MOPD (multi-teacher on-policy distillation): pass more than one entry in teacher_server_urls and route each sample to a teacher via a teacher_id column (e.g. a math teacher and a code teacher, each served independently).

from trl.experimental.async_distillation import AsyncDistillationConfig, AsyncDistillationTrainer

trainer = AsyncDistillationTrainer(
    model="Qwen/Qwen3-4B",
    args=AsyncDistillationConfig(
        teacher_server_urls=["http://teacher-math:8000", "http://teacher-code:8000"],
        teacher_top_k=16,
    ),
    train_dataset=dataset,  # rows carry a `teacher_id` column for MOPD
)
trainer.train()

by @kashif in #6705

New model & template support

Examples reorganized into per-example folders

examples/ was split by format (scripts/ vs notebooks/), which scattered related files (e.g. sft_nemotron_3 existed as both a script and a notebook, in different folders; OpenEnv notebooks lived apart from OpenEnv scripts). Everything is now one folder per story — the name says the method and the task (grpo_wordle, sft_gpt_oss, ppo_tldr), and the folder holds every file the example needs (scripts, notebooks, prompts, chat templates, eval code). Thin single-trainer demo scripts are removed (they live in trl/scripts/ and each trainer doc page has a runnable snippet). Old examples/scripts/… and examples/notebooks/… GitHub / Colab links will 404 — worth flagging in downstream posts.

by @qgallouedec in #6820

AsyncGRPO: metrics rework, dtype control, and cleanup

Rewritten TRL skill (Python-API first) + drop trl skills install

The TRL skill is now written around the Python API — trainer selection, dataset format, load-bearing config fields, GRPO reward-function contract — with the CLI as a footnote. trl skills install is dropped and the skill source moves out of the package.

DistillationTrainer supports tool calling

Continuing the graduation cycle from v1.10.

by @qgallouedec in #6723

Suggest migrating to DistillationTrainer when the GKD config is fully covered

GKDTrainer now emits a hint when the config is fully covered by the stable DistillationTrainer — no behavior change, just a nudge for users still on the older API.

by @qgallouedec in #6847

Other

vLLM version sweep

Fixes

  • Fix GRPO/RLOO training with multiple images for VLMs without grid metadata by @qgallouedec in #6570
  • Fix GRPO/RLOO with dynamic-tiling VLMs (InternVL) by @qgallouedec in #6619
  • Handle None token logprobs from vLLM in GRPO importance sampling by @behroozazarkhalili in #6693
  • [GRPO] Match the Liger dapo / cispo / vespo normalizer to the non-Liger path — completes the fix from v1.10 (#6024) on the Liger path. By @kashif in #5890
  • [SFT] Treat past_key_values and attentions as optional on the model output by @Hakureirm in #6686
  • Fix AsyncGRPOTrainer crashing on environment-owned datasets by @albertvillanova in #6824
  • Fix SDPO scale_rewards CLI parsing and warn on ignored feedback by @sergiopaniego in #6341
  • Fix opencode.py proxy trace / log paths after OpenEnv moved them to functions by @sergiopaniego in #6672

Documentation and Examples

  • Rework AsyncGRPO docs alongside the metrics rework (@AmineDiro, #6715)
  • Reorganize examples into self-contained per-example folders (see above)
  • Suggest migrating to DistillationTrainer when GKD config is covered (see above)

CI

Big CI-hygiene cycle by @albertvillanova. Highlights:

New Contributors

What's Changed

Full Changelog: v1.10.0...v1.11.0

Contributors

kashif, albertvillanova, and 7 other contributors
Assets 2
Loading