Skip to content

v1.10.0

Latest
Latest

Choose a tag to compare

View all tags
@qgallouedec qgallouedec released this 13 Aug 01:36
· 2 commits to main since this release
v1.10.0
a7be897
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

🎓 DistillationTrainer is now a stable trainer

After a ~30-PR refactor that reshaped its data contract, loss path, generation stack, config surface, and tests, DistillationTrainer and DistillationConfig graduate from trl.experimental.distillation to the top-level trl package. Same import surface as SFT / DPO / GRPO / KTO. The old experimental path still works and emits a FutureWarning (removal in v2.0.0).

# Before
from trl.experimental.distillation import DistillationConfig, DistillationTrainer

# Now
from trl import DistillationConfig, DistillationTrainer

Also lands a trl distillation CLI and moves the tests to tests/. The full refactor arc: switched signature columns to prompt (deprecated messages-format), pinned the generation stack to GRPO's, wired the chunked JSD loss and deleted the full-logit path, cleaned the Liger path to share extraction with the chunked path, adopted GRPO's log() / training_step timing / _save_checkpoint, rebuilt the test suite to GRPO shape, and much more.

by @qgallouedec across ~30 PRs (#6479, #6480, #6481, #6482, #6484, #6487, #6497, #6508, #6509, #6510, #6511, #6512, #6513, #6521, #6522, #6523, #6524, #6525, #6526, #6530, #6537, #6604, #6605, #6606, #6607, #6609, #6610, #6611, #6612, #6613, #6614, #6629, #6630, #6632, #6633, #6634, #6639, #6640, #6641, #6642, #6643, #6644, #6645, #6647, #6653).

DistillationTrainer supports Vision Language Models

Alongside the graduation, VLMs work end-to-end in DistillationTrainer.

by @qgallouedec in #6650

AsyncGRPO: loop-owning agent training (OpenEnv harness)

New experimental loop-owning (black-box) path for AsyncGRPOTrainer — for training external agents like opencode that run their own tool loop, rather than TRL driving each turn as in the environment_factory white-box path.

The agent runs in an OpenEnv session in transparent_proxy mode; an in-sandbox proxy captures each turn's token ids and logprobs. On completion, TRL reads the trace, rebuilds per-turn training rows, and scores the workspace with the session's verify(). Ships with three caller-supplied policy hooks (rollout_reward_fn, train_turn_fn, agent_turn_fn) so you can drop framework aux calls (title generator, context summarizer) and reinforce only the turns you want.

Includes a self-contained examples/scripts/openenv/opencode.py (local subprocess sandbox + DeepCoder held-out stdin/stdout verifier), validated end-to-end on Qwen3-4B.

by @AmineDiro in #6420, plus HF-sandbox variant by @sergiopaniego in #6565

AsyncGRPO: observability + sampling controls

DiffusionGemma block-diffusion SFT example

A new SFT example for google/diffusiongemma-26B-A4B-it implementing the reference recipe: one response block per step, uniform random token corruption with t ~ U(eps, 1), two-pass self-conditioning, flat cross-entropy over the whole canvas plus an autoregressive co-loss on the encoder. LoRA targets attention + dense MLP linears. Requires transformers >= 5.12.0.

Ships with diffusion_gemma.jinja / diffusion_gemma_training.jinja chat templates (with {% generation %} markers) so assistant_only_loss=True works out of the box.

by @kashif in #6003

More model & template support

Default changes

Two default flips this cycle — pinning them explicitly is recommended if you want the old behavior.

  • SFTConfig.max_completion_length / GRPOConfig.max_completion_length: default bumped from 256 to 512. By @dhruvnigam93 in #6264
  • GRPOConfig.use_bias_correction_kl now defaults to True. By @gowtham-sai-yadav in #6503

Packing and padding-free on VLMs with text-only data

VLM configurations no longer reject packing / padding-free when the batch happens to be text-only.

by @cris96spa in #6547

Other

vLLM version sweep

Fixes

  • Fix DAPO / CISPO / VESPO loss normalization when steps_per_generation != gradient_accumulation_steps — gradients were silently mis-scaled by gradient_accumulation_steps / steps_per_generation (e.g. 0.5× or 4× the intended token-mean in common configs). Default-equivalent configs are unaffected. By @0xadvait in #6024
  • Fix Liger kernel crash on pre-Ampere GPUs by @albertvillanova in #6518
  • Fix vLLM weight synchronization when vllm_enable_sleep_mode=True by @muupan in #5313
  • Fix vLLM server-mode communicator initialization to use the current accelerator device by @sywangyi in #6417
  • Use accelerator device instead of hardcoded cuda in FSDP2 vLLM weight sync by @verma8076 in #6592
  • Fix prepare_deepspeed crash with CPU offload optimizer by @roycho96 in #5916
  • fix(ppo): exclude padding tokens from entropy calculation by @mukund1985 in #6121
  • fix: avoid CopySlices when scaling policy logits by @DaoyuanLi2816 in #6554
  • Fix multimodal field handling in GRPO tool-call loop by @HariomJangra in #6286
  • [GRPO] Apply the completion mask elementwise in the LuSPO loss aggregation by @YaseenBashaT in #6654
  • [GRPO] Fix entropy bonus normalization inconsistency across loss types by @YaseenBashaT in #6648
  • Fix incorrect example script path in Online DPO docs by @latent-9 in #6598
  • Point the opencode example at huggingface/OpenEnv by @sergiopaniego in #6529
  • Remove symlink support from skills installation by @albertvillanova in #6665

Documentation and Examples

CI

New Contributors

What's Changed

  • ⬆️ Bump dev version by @qgallouedec in #6496
  • Test train_dataset=None raises for core trainers by @albertvillanova in #6492
  • Add regression tests for evaluating init-time eval datasets after training with precomputed reference log-probs by @albertvillanova in #6488
  • docs: complete GRPOConfig loss_type help by @gowtham-sai-yadav in #6477
  • test: don't hard-code bf16=True on devices that lack bf16 support by @behroozazarkhalili in #6036
  • [DistillationTrainer refactor] Switch tests, docs, and example to prompt-only datasets by @qgallouedec in #6479
  • cleanup xpu cache memory after each test by @kaixuanliu in #5771
  • Fix vLLM server-mode communicator initialization to use the current A… by @sywangyi in #6417
  • [DistillationTrainer refactor] Remove messages-format support and prompt-length config by @qgallouedec in #6480
  • [DistillationTrainer refactor] Pin signature columns to ["prompt", "image", "images"] by @qgallouedec in #6481
  • [DistillationTrainer refactor] Emit completion_mask alongside labels by @qgallouedec in #6482
  • [DistillationTrainer refactor] Loss consumes completion_mask by @qgallouedec in #6484
  • Fix : queue wait time metric AsyncGRPOTrainer by @AmineDiro in #6489
  • [DistillationTrainer refactor] Emit prompt_ids / prompt_mask / completion_ids by @qgallouedec in #6487
  • Remove unnecessary loss_type in KTO precompute ref log-probs test by @albertvillanova in #6501
  • Force use_cpu in tests when no CUDA device is available by @albertvillanova in #6504
  • Hotfix CI: Temporarily pin liger-kernel < 0.8.1 by @albertvillanova in #6517
  • Silence transformers LOAD REPORT tables in test output by @albertvillanova in #6502
  • Change default max_completion_length from 256 to 512 by @dhruvnigam93 in #6264
  • Add DiffusionGemma block-diffusion SFT example by @kashif in #6003
  • Record rollout traces to trackio by @AmineDiro in #6491
  • [DistillationTrainer refactor] Switch the loss to GRPO's keys by @qgallouedec in #6497
  • [DistillationTrainer refactor] Freeze ServerDistillationTrainer self-contained by @qgallouedec in #6508
  • Add LFM2 and LFM2.5 support and testing by @qgallouedec in #6428
  • [DistillationTrainer refactor] Delete _compute_prompt_length by @qgallouedec in #6509
  • Async grpo OpenEnv harness rollout by @AmineDiro in #6420
  • [DistillationTrainer refactor] Add _tokenize_prompts (unwired) by @qgallouedec in #6510
  • [DistillationTrainer refactor] Add the shared generation config fields (add-only) by @qgallouedec in #6511
  • Fix crash when training with Liger kernel on pre-Ampere GPUs by @albertvillanova in #6518
  • [DistillationTrainer refactor] Add the ctor generation-supporting state (add-only) by @qgallouedec in #6512
  • Fix prepare_deepspeed crash with cpu offload optimizer by @roycho96 in #5916
  • Fix CI quality: Fix copyright year in trl/experimental/async_grpo/openenv_harness.py by @albertvillanova in #6531
  • Reject unsupported train_dataset types in core trainers by @albertvillanova in #6493
  • [DistillationTrainer refactor] Add _generate_single_turn (unwired) by @qgallouedec in #6513
  • [DistillationTrainer refactor] Add _generate + _generate_and_score_completions (unwired) by @qgallouedec in #6521
  • Test train_dataset types at init for core trainers by @albertvillanova in #6494
  • Add Gemma4 response template by @qgallouedec in #6535
  • Validate that reward functions return one reward per completion by @gowtham-sai-yadav in #6534
  • fix(grpo): correct DAPO/CISPO/VESPO loss normalization when steps_per_generation != gradient_accumulation_steps by @0xadvait in #6024
  • Hotfix CI: Temporarily pin bitsandbytes < 0.50.0 by @albertvillanova in #6544
  • Hotfix CI: Xfail NemotronH GRPO/RLOO tests against a broken transformers dev build by @albertvillanova in #6542
  • added step time metric to AsyncGRPOTrainer by @AmineDiro in #6490
  • Allow packing and padding-free on VLMs with text-only data by @cris96spa in #6547
  • fix(ppo): exclude padding tokens from entropy calculation by @mukund1985 in #6121
  • Respect TQDM_DISABLE in DPO/KTO/BCO reference log-prob loops by @qgallouedec in #6507
  • Revert xfail for NemotronH GRPO/RLOO tests now that transformers#47569 fixed the kernels bug by @albertvillanova in #6558
  • fix: avoid CopySlices when scaling policy logits by @DaoyuanLi2816 in #6554
  • Fix a bug in vLLM weight synchronization when vllm_enable_sleep_mode=True by @muupan in #5313
  • Point the opencode example at huggingface/OpenEnv by @sergiopaniego in #6529
  • Bump the actions group with 3 updates by @dependabot[bot] in #6540
  • Suppress deprecated torch.get_autocast_gpu_dtype() warning from mamba-ssm kernel by @albertvillanova in #6556
  • Fix test_peft_with_quantization tests for bitsandbytes 0.50.0 by @albertvillanova in #6546
  • Fix patch release instructions: Pull release branch instead of main by @albertvillanova in #6563
  • Fix inaccuracies in the release instructions by @albertvillanova in #6566
  • Use dtype instead of torch_dtype (deprecated) by @qgallouedec in #6560
  • Fix test: don't call add_response_schema when the model already ships a response template by @qgallouedec in #6574
  • [DistillationTrainer refactor] Switch generation to GRPO's stack; delete the buffer by @qgallouedec in #6522
  • [DistillationTrainer refactor] Align the generation_kwargs dict with GRPO by @qgallouedec in #6523
  • [DistillationTrainer refactor] Align the vLLM ctor block with GRPO by @qgallouedec in #6524
  • [DistillationTrainer refactor] Add _get_last_hidden_state (unwired) by @qgallouedec in #6525
  • Default use_bias_correction_kl=True in GRPOConfig by @gowtham-sai-yadav in #6503
  • Ignore benign bitsandbytes 4-bit blocksize alignment warning in CI by @albertvillanova in #6582
  • Respect use_cpu in create_model_from_path device_map default by @Strongich in #6295
  • Ignore transformers rotary device deprecation warning in CI by @albertvillanova in #6585
  • Add test_train_moe_peft_model for KTO by @albertvillanova in #6589
  • Revert transformers rotary device deprecation warning filter by @albertvillanova in #6596
  • Document removal criteria for pytest warning filters by @albertvillanova in #6595
  • Reject unsupported f_divergence_type / loss type combinations in DPO by @qgallouedec in #6559
  • Add support for vLLM 0.26.0 by @qgallouedec in #6569
  • Use a reference adapter with target_parameters on peft>=0.20.0 by @albertvillanova in #6591
  • Fix incorrect example script path in Online DPO docs by @latent-9 in #6598
  • Add top_p/top_k/min_p/repetition_penalty to AsyncGRPOConfig by @AmineDiro in #6608
  • Use accelerator device instead of hardcoded cuda in FSDP2 vLLM weight sync by @verma8076 in #6592
  • [DistillationTrainer refactor] Add the chunked JSD loss (unwired) by @qgallouedec in #6526
  • [DistillationTrainer refactor] Wire compute_loss to the chunked JSD path; delete the full-logit loss by @qgallouedec in #6530
  • [DistillationTrainer refactor] Heavy-clean the Liger path: share extraction with the chunked path by @qgallouedec in #6537
  • [DistillationTrainer refactor] Default disable_dropout to False by @qgallouedec in #6604
  • [DistillationTrainer refactor] Config finish by @qgallouedec in #6605
  • [DistillationTrainer refactor] Constructor cleanup: safe internal cleanups by @qgallouedec in #6606
  • [DistillationTrainer refactor] Config cosmetic parity by @qgallouedec in #6607
  • [DistillationTrainer refactor] Constructor cleanup (40b): drop eval-metric params, require model by @qgallouedec in #6609
  • [DistillationTrainer refactor] Constructor cleanup (40c): quantization + PEFT/QLoRA parity by @qgallouedec in #6610
  • [DistillationTrainer refactor] Make advantages optional in print_prompt_completions_sample by @qgallouedec in #6611
  • [DistillationTrainer refactor] Use the shared print_prompt_completions_sample by @qgallouedec in #6612
  • [DistillationTrainer refactor] Adopt GRPO's log() + bounded _logs deques by @qgallouedec in #6613
  • [DistillationTrainer refactor] Adopt GRPO's training_step timing + _save_checkpoint by @qgallouedec in #6614
  • Require vision for VLM response parsing tests by @albertvillanova in #6602
  • Require jmespath only on transformers below 5.13.0 by @albertvillanova in #6597
  • Add missing logged metrics to the GRPO and RLOO docs by @PoojanTa in #6624
  • Close in-flight harness sessions concurrently on shutdown by @sergiopaniego in #6564
  • Add opencode_hf_sandbox.py: loop-owning training in remote HF sandboxes by @sergiopaniego in #6565
  • Remove symlink support from skills installation by @albertvillanova in #6665
  • ci: exclude the buggy TruffleHog lob detector by @behroozazarkhalili in #6674
  • [GRPO] Apply the completion mask elementwise in the luspo loss aggregation by @YaseenBashaT in #6654
  • [GRPO] Fix entropy bonus normalization inconsistency across loss types by @YaseenBashaT in #6648
  • Validate YAML config root type by @RerankerGuo in #6656
  • Fix multimodal field handling in GRPO tool call loop by @HariomJangra in #6286
  • [DistillationTrainer refactor] Reject use_liger_kernel with logit_scale / final_logit_softcapping by @qgallouedec in #6629
  • [DistillationTrainer refactor] Convention/cosmetic sweep by @qgallouedec in #6630
  • [DistillationTrainer refactor] Config: GRPO parity drift fixes by @qgallouedec in #6632
  • [DistillationTrainer refactor] Log the student entropy metric by @qgallouedec in #6633
  • [DistillationTrainer refactor] Rewrite the test suite to GRPO shape by @qgallouedec in #6634
  • [DistillationTrainer refactor] Docs: align distillation_trainer.md with stable-trainer conventions by @qgallouedec in #6639
  • [DistillationTrainer refactor] Promote DistillationTrainer to the stable API by @qgallouedec in #6640
  • [DistillationTrainer refactor] Move DistillationTrainer to the Stable telemetry group by @qgallouedec in #6641
  • [DistillationTrainer refactor] Add the trl distillation CLI by @qgallouedec in #6642
  • [DistillationTrainer refactor] Move the distillation tests to tests/ by @qgallouedec in #6647
  • [DistillationTrainer refactor] Docs: de-namespace DistillationTrainer for the stable API by @qgallouedec in #6643
  • [DistillationTrainer refactor] Update the example script for the stable API by @qgallouedec in #6644
  • [DistillationTrainer refactor] Re-point ServerDistillationTrainer at the stable base by @qgallouedec in #6645
  • [DistillationTrainer refactor] Support Vision Language Models by @qgallouedec in #6650
  • [DistillationTrainer refactor] Docs: announce the DistillationTrainer promotion in What's New by @qgallouedec in #6653
  • Release: v1.10 by @qgallouedec in #6724

Full Changelog: v1.9.0...v1.10.0

Contributors

kashif, muupan, and 22 other contributors
Assets 2
Loading