SkyRL: v0.3.0

Highlights

HTTP-based inference

We introduced a new HTTP-based inference architecture in v0.2.0. This simplifies inference engine management, allows us to integrate high performance routers like vllm-router and supports advance serving patterns like prefill-decode disaggregation. The legacy inference codepath has now been removed in v0.3.0.

Improved load balancing

By default, SkyRL uses a fixed session ID per trajectory and maps a given session ID to a given vLLM engine replica with consistent-hashing. This maximizes KV cache reuse for a given trajectory. However, simple session-based routing can be suboptimal for overall generation throughput in async RL. We’ve implemented a custom routing policy with vllm-router called sticky_least_loaded that load balances trajectories across replicas with sticky routing per trajectory. On our benchmarks for fully async RL on a multi-turn agentic task (averaging 40+ turns) with a large MoE model (GLM 4.7), this load balancing improves throughput by upto 1.4x. See #1791 for more details.

Dynamic microbatching

We added support for dynamic microbatching to improve training performance. Dynamic microbatching packs sequences in each worker upto a fixed token count, instead of a fixed number of sequences per forward pass. This balances real token counts across microbatches, improving throughput and memory utilization when sequence lengths are highly variable. Training throughput improvements are dependent on the task as well as the model and cluster shape, but we see gains of 2x or more common math and agentic tasks. See: https://docs.skyrl.ai/docs/api-ref/skyrl/config#attr-max_tokens_per_microbatch for more details

Dynamic sampling

We’ve added support for dynamic sampling in fully async RL with a new trainer.fully_async.sample_full_batch flag. When enabled, SkyRL drops zero variance groups (groups where all the trajectories score the same reward) and continues sampling until the training mini batch is filled. Zero-variance groups can negatively affect learning and destabilize training. While these zero variance groups receive zero advantages with GRPO, they still affect learning by biasing the aggregate mini-batch loss. Dynamic sampling ensures that such groups are filtered out while keeping a consistent batch size during optimization steps. See: https://docs.skyrl.ai/docs/api-ref/skyrl/config#attr-sample_full_batch for details on usage.

Cache salt for stale KV blocks

SkyRL now defaults to adding a cache salt based on the current model version to avoid re-using stale KV blocks, improving stability during async RL. See: https://docs.skyrl.ai/docs/api-ref/skyrl/config#attr-use_cache_salt for usage

(Experimental) Multi-LoRA support with Tinker

We’ve implemented initial support for multi-LoRA training with the Tinker API. Multi-LoRA training enables researchers to share compute across experiments with a warm, multi-tenant Tinker server. Multi-tenant training can reduce GPU memory requirements by sharing a fixed base model with multiple LoRAs on the trainer and inference engine. Since generations are typically long tailed, multi-tenancy also improves GPU utilization by overlapping requests from concurrent experiments and reduces overall job completions compared to sequential training. For 8 concurrent experiments with Qwen3-4B-Instruct-2507 on a single H200 node, multi-LoRA training can provide E2E speedups of 2.8x. Multi-LoRA support in SkyRL has been co-developed with Trajectory AI, who have shared more results in their blog post.

Native SFT support

SkyRL supports a new SFTTrainer for native SFT training, providing researchers a unified post-training library for SFT and RL. Features added:

• Diverse dataset formats: SFTTrainer supports instruction following and conversational datasets, including support for tool messages.
• Sequence packing: Sequence packing packs the global training batch into bins to reduce wasted computation on padding tokens, improving throughput in some settings by 2.75x.
• Custom callbacks and custom sampling support: Custom callbacks and custom sampling support allow users to modify almost any part of the training loop - bringing data mixtures, using curriculum sampling, logging custom metrics, and more.
• VLM SFT: Initial support for SFT on VLM models with the Megatron backend, contributed by Suhas!

(Experimental) Speculative decoding with MTP

SkyRL now has experimental support for Multi-Token Prediction (MTP): train a model’s native MTP heads and use them during generation to improve inference throughput. MTP training is implemented in the Megatron backend, by adding a draft loss that trains the MTP heads on the next token prediction task in a decoupled way from the policy head. The updated MTP heads are synced along with the rest of the policy model weights during weight transfer. Speculative decoding with MTP improves generation throughput by upto 2x for Qwen/Qwen3.5-9B on the DAPO recipe (synchronous RL) by reducing tail latency for stragglers. Currently, we only support MTP training with the Megatron backend. We are actively working on benchmarking for multi-turn and async RL scenarios. See: https://docs.skyrl.ai/docs/examples/spec_decode for more details

(Experimental) INT4 fake QAT training

We’ve integrated experimental support for INT4 Quantization-Aware training (QAT). INT4 quantization enables serving extremely large models like Kimi K2.6 (1T+ parameters) on a single 8xH200/ 8xB200 node. However, using INT4 precision for rollouts but BF16 for training can increase trainer <> inference mismatch and destabilize training. INT4 “fake” QAT works by emulating INT4 computation during training by discretizing the BF16 weights into INT4 (in floating point) and then restoring them. We’ve added support for INT4 QAT training with LoRA as an initial step in supporting large models like Kimi K2.6. See the configuration guide: https://docs.skyrl.ai/docs/api-ref/skyrl/config#attr-fake_int4_qat for more details. Thanks to Casper for the contribution, and to Miles for the original work!

What's changed

• [chore] Move flash attention wheel specification to uv.sources by @SumanthRH in #1546
• [tinker] Fix optimizer offload in SkyRL train backend after weight sync by @SumanthRH in #1550
• [bug][tinker] Fix vLLMRouter init in SkyRLTrainBackend by @SumanthRH in #1551
• [bug] Fix failing CPU tests after #1544 by @SumanthRH in #1555
• [train] Support packing for CUDA IPC transfer with new inference codepath by @SumanthRH in #1557
• support trajectory-based agents and OpenHands (#1184) by @arteemg in #1548
• Update skyrl and skyrl-gym versions after 0.2.0 release by @SumanthRH in #1562
• [chore] Redirect output to infrastructure logs for VLLMServerActor by @SumanthRH in #1563
• [harbor][step-wise] Make Harbor use step-wise training by @CharlieFRuan in #1542
• [docs][example] LoRA config for Geometry3k by @nithinvc in #1569
• [tinker] Fix single request batching in TinkerEngine by @pcmoritz in #1489
• [skyrl][tinker] Multi-modal Tinker Sampling by @nithinvc in #1484
• [feat][megatron] Add freeze_moe_router config to skip router parameters by @SumanthRH in #1570
• [bug] Fix megatron LoRA with new inference codepath by @SumanthRH in #1572
• [tinker] Make it clear that a snapshot with persist=false is a marker by @pcmoritz in #1565
• [tinker] Support PPO loss with Tinker and add critic model in SkyRLTrainBackend by @tamoghnokandar in #1389
• [train] LoRA-only weight syncing for Megatron by @hao-aaron in #1360
• added log_example_interval by @agolajko in #1522
• [ci] Add CI for supported models for megatron backend, and add supported model page to docs by @erictang000 in #1508
• [tinker] add /api/v1/client/config POST endpoint by @erictang000 in #1578
• [ci] skip megatron deps for non-megatron CI, disable sample packing for MLA on L4 by @erictang000 in #1585
• [qoc][trainer] Use WorkerDispatch for weight sync instead of policy_model by @CharlieFRuan in #1584
• [deps] Bump megatron-bridge to latest by @erictang000 in #1586
• [bug][tinker] Fix data_parallel_size > 1 in SkyRLTrainBackend by @SumanthRH in #1580
• [fix][train] Fix weight metadata handling for Megatron backend with native vLLM weight sync by @SumanthRH in #1581
• [CI] Reorg Megatron metadata consistency test by @SumanthRH in #1587
• [qoc][trainer] Remove unused PPORayActorGroup and Dispatch methods by @CharlieFRuan in #1588
• [harbor] Use old inference codepath for harbor by @CharlieFRuan in #1593
• [qoc] Remove dead code XXXWorkerBase.barrier() by @CharlieFRuan in #1594
• [qoc][trainer] Remove dead code in workers by @CharlieFRuan in #1595
• [qoc][worker] Deduplicate checkpoint and lr methods by @CharlieFRuan in #1598
• [cleanup] Remove NCCL_CUMEM_ENABLE=0 from prepare_runtime_environment by @erictang000 in #1600
• [qoc][workers] Unify offload/onload methods across roles and backend by @CharlieFRuan in #1602
• [bug] Fix AttributeError on MambaLayer in freeze_moe_router by @SumanthRH in #1608
• [bug] Fix LoRA disk reloading and sleep issues by @hao-aaron in #1606
• [fix] Add trailing slash to local directory for uploads to existing cloud directory with fsspec by @SumanthRH in #1614
• [bug] Lora bug fixes + port collision fix by @hao-aaron in #1625
• [examples] Add qwen3.5-35-a3b DAPO script by @erictang000 in #1627
• [train] SFT 2/N: Support training on all assistant messages, add HF export support and num_epochs by @SumanthRH in #1611
• [feat] Multi-LoRA serving for RemoteInferenceClient by @hao-aaron in #1579
• Add FP8 text-to-SQL training script; drop SQLEnv stop-tag assertion by @hao-aaron in #1599
• [vllm] revert removal of NCCL_CUMEM_ENABLE=0 by default in prepare_runtime_environment by @erictang000 in #1629
• [train] Default to chunked logprobs for Megatron by @SumanthRH in #1610
• [fix] Fix old inference generate path to take in model as an optional Arg by @erictang000 in #1631
• [tinker][megatron] Multi-LoRA Megatron + Tinker API by @erictang000 in #1617
• [feat] Recursive language models (RLMs) environment by @sox8502 in #1596
• Fix nan leak from masked positions in compute_approx_kl by @jamesbraza in #1635
• [fix] use token_mean_legacy loss reduction for DAPO scripts by @SumanthRH in #1637
• [tinker][megatron] Multi-LoRA Megatron + Tinker API RL Training by @erictang000 in #1621
• [tx] Add initial implementation of RayJaxBackend by @andrewsykim in #1418
• [megatron] Fix failing CI test for megatron chunked logprobs + entropy loss by @erictang000 in #1648
• [megatron] Forward adam_betas in init_megatron_optim_config by @erictang000 in #1649
• [dependencies] Bump vllm to 0.20.1, torch to 2.11 by @erictang000 in #1628
• [examples] Add a simple fully async RL script to demonstrate async RL with new inference by @SumanthRH in #1651
• [feat] Add prometheus logging to new inference, add vllm metrics to wandb by @hao-aaron in #1636
• [docs] Add architecture page for HTTP-based inference in SkyRL by @SumanthRH in #1655
• [CI] Add E2E CI for Tinker API with SkyRLTrainBackend by @hao-aaron in #1616
• [cleanup] Remove FSDP1 support + make 'fsdp' default to fsdp2 by @erictang000 in #1659
• Add alphaxiv RLM project to README.md by @erictang000 in #1661
• [tinker] Forward sample requests directly to backend vLLM (non-colocated) by @erictang000 in #1638
• [lora][tinker] Add pause and resume for multi-tenant lora by @hao-aaron in #1657
• [examples] Change default correction to be geometric sequence masking by @SumanthRH in #1665
• [feat] Add a Tinker-compatible forward(loss_fn=...) API to SkyRL's FSDP and Megatron policy workers and refactor worker outputs by @SumanthRH in #1667
• [CI] add E2E CI for SFT on the Tulu3 dataset with the Megatron backend by @SumanthRH in #1670
• [feat] Add support for evaluation dataset in SFTTrainer by @SumanthRH in #1668
• [models] Add nemotron-nano-30b-a3b to CI by @erictang000 in #1603
• ThunderAgent integration and R2Egym code agent rl training receipe. by @HaoKang-Timmy in #1645
• [build] Update Megatron Dockerfile after torch 2.10 upgrade and transformer-engine upgrade by @SumanthRH in #1671
• [examples] Fix DAPO preprocessing script by @SumanthRH in #1676
• Revert "[lora][tinker] Add pause and resume for multi-tenant lora " by @erictang000 in #1678
• [train] Fix chunked-logprob backward and skip chunking for short sequences by @SumanthRH in #1650
• [multi-lora] skip pause/resume for lora with merge=false by @erictang000 in #1677
• [feat] Support tool-calling datasets in SFTTrainer by @sdesai-lila in #1675
• [chore] Add CLAUDE.md and .claude files, improve troubleshooting guide by @SumanthRH in #1682
• [bug] Fix documentation build issue after #1682 by @SumanthRH in #1683
• [chore] Cleanup legacy YAMLs in SkyRL by @SumanthRH in #1684
• [chore] Add epoch and global_step as metrics for SFTTrainer by @SumanthRH in #1693
• feat: Optimize SFT dataloader with slice-based tokenization and caching by @jlee-lila in #1695
• [chore] Set a default python version of 3.12 by @SumanthRH in #1710
• [logging] Add MoE auxiliary loss by @erictang000 in #1705
• [logging] Wrap run in try/except + surface exceptions to wandb if supported by @erictang000 in #1706
• [metrics] track min/max logprob diff in trainer by @erictang000 in #1707
• [megatron] add aux loss coeff explicitly and add post init to keep config default kwargs even when overriden by @erictang000 in #1708
• [logging] add tokens/sec/gpu as a metric tracked in the trainer by @erictang000 in #1711
• Removed dead timeout parameter from DistributedStrategy subclasses by @jamesbraza in #1699
• GPU utilization from Ray for both RL + SFT by @erictang000 in #1712
• [feat] Support training callbacks in SkyRL by @SumanthRH in #1696
• fix: prevent process hang when vLLM engine dies during async generation by @dinhxuanvu in #1691
• [fix] Explicitly set vLLM device_type to cuda by @SumanthRH in #1724
• [tinker] Add deterministic routing support for sample requests by @SumanthRH in #1720
• [tinker] Add session-based routing example + docs by @SumanthRH in #1730
• [megatron] make empty_cuda_cache default true for megatron by @erictang000 in #1731
• Fix meta-init bf16 cast corrupting rotary inv_freq under sequence parallelism by @jamesbraza in #1725
• [feat] Add prompt_mean loss reduction by @erictang000 in #1719
• vllm metrics docs by @hao-aaron in #1662
• [bug] Switch to reload_weights API for loading weights in legacy inference codepath by @SumanthRH in #1685
• [ci] H100 CI by @hao-aaron in #1679
• [config] Rename use_sample_packing to remove_microbatch_padding by @SumanthRH in #1738
• Fix FSDP Tinker backend crash on model creation by @erictang000 in #1739
• [tinker][docs] update tinker API docs to use qwen3-4b-instruct-2507 by @erictang000 in #1740
• [docs] Fix prod build: use text instead of unknown promql code fence by @hao-aaron in #1736
• [CI] Add anyscale cloud for H100 CI by @SumanthRH in #1745
• [feat] Add support for sequence packing with SFTTrainer by @SumanthRH in #1734
• [CI] Bump timeout for Megatron GPU CI by @SumanthRH in #1747
• [CI] Revert back to transformers 5.2.0 for Qwen 3.5 CPU test by @SumanthRH in #1748
• [tinker] Support prompt_logprobs in SkyRLTrainBackend sample() path by @pbokc in #1461
• [tinker] Fix checkpointing for multi-node runs by @pcmoritz in #1732
• [megatron] compute logprobs/entropy on CP sharded logits + fix CP loss scaling by @jinghanyao1-hub in #1751
• [CI] Use the right cloud for H100 CI by @SumanthRH in #1749
• [chore] bump megatron core to latest dev branch by @erictang000 in #1754
• [train] Use custom wheel for vllm-router for /chat/completions fix by @SumanthRH in #1601
• [chore] Exclude nixl-cu13 from packages by @SumanthRH in #1756
• [chore] pin nixl-cu12 by @erictang000 in #1759
• [ci] add commit names to gpu ci jobs to avoid collisions with anyscale job wait by @erictang000 in #1760
• [examples] Reduce train batch size for GLM 4.7 30B script by @SumanthRH in #1761
• [ci] fix CI bugs introduced after latest uv sync by @erictang000 in #1763
• Log and document that colocate_all=false still colocates policy and ref by @jamesbraza in #1758
• [chore] move megatron-core from dev latest to main latest, bump megatron-bridge to latest by @erictang000 in #1762
• [docs] Add docs for session based routing by @SumanthRH in #1764
• Make prompt dataloader num_workers/persistent_workers configurable by @jamesbraza in #1746
• feat(tracking): plumb wandb tags through Tracking init by @dinhxuanvu in #1698
• [fix] Save trust_remote_code artifacts in MegatronStrategy.save_hf_model by @dinhxuanvu in #1726
• [fix] Preserve original AutoConfig in MegatronWorker.init_configs by @dinhxuanvu in #1727
• [script] Add Qwen3.6-35b LoRA DAPO script by @CharlieFRuan in #1775
• [chore] Update harbor commit after session-id routing fix by @SumanthRH in #1774
• [tinker] Pin tinker SDK back to 0.16.1 to fix CPU test hang by @pcmoritz in #1779
• [tinker] Implement get_sampler API by @pcmoritz in #1773
• [chore] Add aarch64 wheel for vllm-router by @SumanthRH in #1781
• [perf] Add expandable_segments controls for trainer and inference engine by @erictang000 in #1768
• [tinker] Unpin tinker SDK, bump to 0.22.3 by @pcmoritz in #1780
• feat: add max_tokens_per_microbatch config for token-based micro-batching by @erictang000 in #1477
• [CI] Allow non-committer PRs to run with labels by @SumanthRH in #1786
• Fix silent MoE corruption in the legacy multi-chunk weight sync by @jamesbraza in #1737
• [fix] Switch to nixl package and exclude nixl-cu13; Fix cleanup in test_inference_server_group by @SumanthRH in #1788
• [ci] revert back to qwen3 after accidental switch to 3.5 for megatron tests by @erictang000 in #1789
• [logging] Flush pending step metrics to the tracker when training crashes by @discobot in #1784
• [ci] move h100 CI to sky cloud by @erictang000 in #1790
• [megatron] Add seq packing support for qwen3.5 by @erictang000 in #1769
• [ci] CI fixes for #1769 by @erictang000 in #1795
• [ci] bump all workflows to anyscale 0.26.103 for consistency by @erictang000 in #1796
• [ci] wait correctly for h100 ci by @erictang000 in #1797
• feat: add max_training_steps config for CI smoke tests by @dinhxuanvu in #1690
• [algorithm] add DPPO policy loss type by @jonahsamost in #1037
• [async] add flag to toggle whether to clear kv cache after weight sync for fully async training, update default max_staleness_steps to 0 by @erictang000 in #1798
• [feat] Add support for sticky_least_loaded router policy with vllm-router by @SumanthRH in #1791
• [ci] fix h100 ci by skipping ddp init for inference only for megatron by @erictang000 in #1803
• [Fix] Fix race condition with memory consumption by using ray_init_fixture by @SumanthRH in #1787
• [feat] Add a serve entrypoint to serve inference endpoints by @SumanthRH in #1799
• [deps] cap fastapi<0.137 to fix vLLM /health 500 (_IncludedRouter) by @erictang000 in #1810
• [fully_async] add sample_full_batch: drop zero-variance groups and refill the mini-batch by @erictang000 in #1802
• [Fix] Add rollout_logprobs key to padding batches with dynamic micro batching by @SumanthRH in #1815
• [feat] Add a fully async trainer simulation script for iterating on generation settings by @SumanthRH in #1814
• [fix] Include masked out batches in metrics calculation with max_token_per_microbatch > 0 by @SumanthRH in #1817
• [docs] fix typo by @kyuds in #1820
• [chore][logging] Add trajectory and group completion metrics for async RL by @SumanthRH in #1804
• [feat] Skip policy fwd logprobs for rollout-based losses (rollout_is, dppo) + worker train/rollout logprob diff metric by @erictang000 in #1813
• [chore] Add tokens_per_turn_max to rollout metrics by @SumanthRH in #1825
• Change enforce_eager default to False by @SumanthRH in #1818
• [fully_async] fix DAPO postprocess_generator_output to accept metrics kwargs by @erictang000 in #1812
• Add reset_running_requests to reset_prefix_cache call for async RL by @SumanthRH in #1824
• [async] add commit=True to early flush wandb logs on fully async trainer by @erictang000 in #1829
• [chore] remove unnecessary def run overrides by @erictang000 in #1830
• [chore]Upgrade vllm to 0.23.0 by @SumanthRH in #1800
• [Fix] vLLM Metrics Scrapper throughput calculation by @zanderjiang in #1794
• feat: Arctic RL training backend integration by @sfc-gh-kganesan in #1837
• [fsdp] Fix redundant downloading of shards across ranks for FSDP checkpointing by @kurtislin in #1844
• [megatron] fused linear cross-entropy for 23x memory savings at 262k context by @casper-hansen in #1841
• gpu ci fix by @hao-aaron in #1849
• [chore] Raise assertion error when policy_loss_type uses recomputed logprobs for fully async RL by @erictang000 in #1850
• fix(generators): compute batched rollout metrics from truncated responses by @EazyReal in #1821
• [fix] Fix num_padding_microbatches metric for Megatron backend by @SumanthRH in #1856
• [refactor] Remove legacy inference codepath by @SumanthRH in #1835
• [megatron] Fix TE sliding-window attention crash at micro_batch>1 on the non-packed forward path by @john-hahn in #1848
• [megatron] Convert fused-MoE expert LoRA to vLLM layout on adapter sync by @casper-hansen in #1857
• [chore] remove mentions of flashrl from the repo and point to vllm quantization support instead by @erictang000 in #1855
• [train] VLM SFT support on Megatron backend (Qwen3-VL) by @s-chundi in #1752
• [chore] fix language_model_only=True case for megatron is_vlm by @erictang000 in #1867
• Make packed sequence alignment FP8-safe by @jinghanyao1-hub in #1828
• [fix] Use current_sampling_params for logprobs by @kyuds in #1868
• [chore] Remove old notice from README by @SumanthRH in #1869
• [megatron] Triton backend for fused LM-head log-prob by @dyurk-lila in #1765
• feat(profiler): drive torch.profiler around the training loop by @dyurk-lila in #1750
• [SFT] Stateful dataloader and custom samplers for SFTTrainer by @avigyabb in #1842
• [deps] bump ray to 2.56.0, add TE wheel to pyproject.toml by @erictang000 in #1872
• [tinker] expose dppo through tinker api by @casper-hansen in #1876
• [train] Save HF processor on checkpoint export for VLMs by @dinhxuanvu in #1785
• [megatron] Accept dtype-string optimizer_config_kwargs (coerce exp_avg_dtype etc. to torch.dtype) by @dyurk-lila in #1805
• [megatron] 1/n towards Kimi K2.6: fake QAT training for Qwen3.6-35B-INT4 by @casper-hansen in #1862
• [tinker] Report rollout/train logprob gap in forward_backward metrics by @casper-hansen in #1880
• [generator] Add per global_step cache salt for each trajectory to invalidate super stale KV by @erictang000 in #1836
• [ci] fix megatron CI block shape error on L4 by @erictang000 in #1889
• add validation sample logging by @erictang000 in #1713
• [bug] fix cpu ci after #1713 by @erictang000 in #1891
• [fix] Avoid inference engine initialization for the SFT code path by @avigyabb in #1878
• [fix] Fix training and evaluation loss metric calculation in SFT codepath by @SumanthRH in #1893
• [chore] Remove "experimental" from fully async RL docs by @SumanthRH in #1904
• [megatron] add opt-in distributed HF checkpoint export by @dinhxuanvu in #1831
• [feat] Add Spec Decoding and MTP training support by @zanderjiang in #1832

New Contributors

• @arteemg made their first contribution in #1548
• @sox8502 made their first contribution in #1596
• @jamesbraza made their first contribution in #1635
• @HaoKang-Timmy made their first contribution in #1645
• @sdesai-lila made their first contribution in #1675
• @jlee-lila made their first contribution in #1695
• @dinhxuanvu made their first contribution in #1691
• @jinghanyao1-hub made their first contribution in #1751
• @discobot made their first contribution in #1784
• @jonahsamost made their first contribution in #1037
• @kyuds made their first contribution in #1820
• @sfc-gh-kganesan made their first contribution in #1837
• @kurtislin made their first contribution in #1844
• @casper-hansen made their first contribution in #1841
• @EazyReal made their first contribution in #1821
• @john-hahn made their first contribution in #1848
• @s-chundi made their first contribution in #1752
• @dyurk-lila made their first contribution in #1765
• @avigyabb made their first contribution in #1842

Full Changelog: skyrl-v0.2.0...skyrl-v0.3.0