Skip to content

Releases

SkyRL: v0.5.0

Choose a tag to compare

@SumanthRH SumanthRH released this 17 Nov 08:58

Summary

SkyRL v0.5.0 continues our work enabling large scale RL on diverse models, with the following key updates:

  • Model Support: Support for the following new models:
    • Beam-1.0 (501B-A23B)
    • DeepSeek-V4.1-Flash
    • Qwen3.8-Flash-Next
  • RL stability
    • Score Centering
  • Scalability
    • Optimizer Offload to Disk
    • 1M-token context for GLM-5.3-Flash
  • New native On-Policy Distillation (OPD) trainer
  • Vision Language Models: sequence packing, context parallelism, and images in agentic RL

In addition, this release updates a number of defaults in SkyRL:

  • Score centering is enabled for compatible loss functions
  • DPPO policy loss
  • Fused LM head

Score Centering

SkyRL now supports score centering (Marek and Ryabinin) to keep RL stable when the rollout and training engines disagree. 4-bit inference for routed experts is an easy way to boost decode throughput on Blackwell, but it creates train-inference mismatch (TIM) that router replay and importance sampling alone can't handle, leading to reward collapse. SkyRL computes the score centering correction exactly over the recorded top-k/top-p sampling support, reusing the packed sample-support path from v0.4.0, without materializing full-vocabulary logits. It works on both Megatron and FSDP.

On an antibody design task with GLM-5.3 (NVFP4 W4A4 rollouts, BF16 trainer), the run without score centering collapsed within 20 steps as TIM and entropy blew up, while the run with score centering kept hill climbing for 80 steps with stable TIM. This integration was contributed by Dominic Yurk from Lila Sciences.

Score centering is automatically enabled when sample-support replay and a compatible sampler-anchored loss are configured. To enable it explicitly:

trainer.algorithm.policy_loss_type=rollout_is
trainer.algorithm.enable_sample_support_replay=true
trainer.algorithm.enable_score_centering=true

PRs: #2484, #2608

Optimizer Offload to Disk

For large MoE models like Beam-1.0-501B-A23B, optimizer state takes ~18 bytes per param (~9 TB in total), which can exhaust host RAM even with CPU offloading. SkyRL now supports offloading optimizer state to disk: each rank stores its optimizer state on node-local NVMe and streams it through the GPU one chunk at a time. Training Beam on 8 B200 nodes, host RAM usage drops from ~13.5 TB to <= 3.2 TB, with each optimizer step taking ~25-50s per rank, comparable to CPU offloading (~35s for similar model sizes).

PR: #2490

Long Context: 1M Tokens for GLM-5.3-Flash

Full fine-tuning of GLM-5.3-Flash can now train on up to 1M tokens per sequence, a 64× increase, through work in four areas:

  • DeepSeek Sparse Attention: GLM's DSA layers now actually run on a sparse kernel (TileLang) instead of silently falling back to dense attention. The DSA indexer is chunked and split across TP ranks.
  • Activation recompute and offloading: full activation recompute, releasing MoE router state after forward, and offloading each layer's saved input to pinned CPU memory.
  • Context parallelism: head-wise (Ulysses-style) CP for GLM's KDA layers, all-gathering the small hidden state instead of all-to-all on projected tensors.
  • Avoiding unnecessary precision upcasts in the mHC projection and MoE combine.

More details in our technical blog.

PRs: #2388, #2411, #2412, #2413, #2414, #2418, #2422, #2433, #2434, #2437

New Model Support

Beam

SkyRL now supports training reflection/Beam-1.0-501B-A23B, building on our Day 0 support effort. LoRA, full fine-tuning and MXFP8 RL are supported out of the box. On the DAPO recipe, all three improve AIME25 avg@8 from ~61% to ~79% in 50 training steps. With R3 enabled, the average logprob diff between Megatron and vLLM stays under 0.01 for LoRA and BF16 full fine-tuning, and under 0.03 for MXFP8 RL.

PRs: #2604, #2618, #2619, #2625

DeepSeek-V4.1-Flash

SkyRL now supports RL on DeepSeek-V4.1-Flash (text only) with the Megatron backend and vLLM inference:

  • LoRA RL with rank-32 adapters on attention and expert layers, synced to vLLM every step. Validated end to end on GSM8K with 8×B200 nodes.
  • Full fine-tuning, with trainer weights re-quantized to FP8/MXFP4 and synced in FP8 without a BF16 round trip.
  • Released checkpoint as is: the trainer loads the released FP8/MXFP4 weights directly, and vLLM serves them with LoRA enabled.

PRs: #2551, #2565, #2566, #2567, #2610, #2612, #2646, #2721

Qwen3.8-Flash-Next

SkyRL now supports Qwen3.8-Flash-Next training on the Megatron backend with both LoRA and full finetuning. We worked with NVIDIA to integrate the cuDNN QSA sparse attention kernels, making training 30% faster than our FlexAttention baseline at 30K tokens with much lower memory. The model scales with expert parallelism (which also shards the frozen 51B-parameter n-gram embedding table) and TP with sequence parallelism, and trains on sequences of up to 262K tokens with context parallelism.

PRs: #2513, #2607

On-Policy Distillation (OPD) Trainer

Building on our previous exploration, SkyRL now has an official on-policy distillation entrypoint and trainer. We implement sampled-token OPD: the teacher grades each token the student samples, giving dense per-token signal.

  • Fewer steps than RL: on DAPO-Math-17k, a Qwen3-4B-Base student matches its DAPO-trained teacher's AIME-2024 accuracy in 20 steps, where DAPO took 90.
  • Minimal teacher overhead: the teacher runs on its own vLLM engines and scores each prompt group as soon as it finishes, overlapping generation and scoring.
  • Combine task rewards with teacher-scored advantages via trainer.algorithm.opd.use_task_reward=true.
  • Managed or self-served teachers: use an existing vLLM endpoint, or let SkyRL launch the teacher.

Coming next: more OPD metrics (#2624, #2632), stabilization strategies (#2622), and multi-teacher OPD (#2679).

Vision Language Models

This release improves VLM training on the Megatron backend for Qwen3-VL and Qwen3.5, and adds images to agentic RL:

  • Sequence packing: VLM sequences are now packed without padding, increasing training throughput by 54% on Geometry-3K with Qwen3.5-9B. Set trainer.remove_microbatch_padding=true.
  • Context parallelism: split long sequences across GPUs with trainer.policy.megatron_config.context_parallel_size=<n> (example script).
  • Images in agentic RL: skycap now captures agent rollouts that include images, so the trainer sees the same image features as vLLM. See the new SWE-bench Multimodal recipe.

PRs:

Changelist

  • [chore] Update version after 0.4.0 release by @SumanthRH in #2390
  • [ci skip][docs] add doc to handoff to agent for new model support by @erictang000 in #2391
  • [deps][megatron] Bump transformer-engine to 2.19.0 by @erictang000 in #2387
  • [megatron] GLM-5.3-Flash DSA: run the fused TileLang SparseMLA kernel by @avigyabb in #2388
  • fix: preserve FIFO trajectory rate admission by @SumanthRH in #2383
  • [deps][megatron] ...
Read more

SkyRL: v0.4.0

Choose a tag to compare

@SumanthRH SumanthRH released this 01 Oct 21:32

In this SkyRL release we focused on improving SkyRL for large scale post-training, with the following key updates:

  • Model Support: Support for the following new models:
    • GLM 5.3 Flash
    • GLM 5.2/5.3
    • Kimi K2.6/2.7
    • Qwen3.8
    • Nemotron 3.5-Lightning
  • RL stability
    • Top-P Sampler Replay
    • Optimized R3 data transfer
    • Full FP8/MXFP8 RL
  • Weight Syncing:
    • Delta Weight Sync
    • Sharded Weight Transfer with Ray Direct Transfer (RDT) + NIXL
  • Scalability
    • Large Scale SFT
    • Tinker Server Scalability

New Model Support

GLM 5.3-Flash and GLM 5.3

SkyRL now supports GLM 5.3 Flash and GLM 5.3 training. We show stable LoRA and full finetuning training with low logprob diff on DAPO with rollout router replay (R3). This enablement effort was in collaboration with Trajectory.

PRs:

Kimi K2.6/2.7

SkyRL now has experimental support for training Kimi 2.6/2.7. Training large 1T+ parameter models is often expensive, requiring 10TB+ of GPU memory even for simple single turn RL tasks. SkyRL now supports efficient training for Kimi 2.6/2.7 models with LoRA, allowing users to train a 1T parameter model on just 2 B300 nodes (see example script here)! We use INT4 serving for inference and apply fake-quantization on the trainer to minimize training inference mismatch.

PRs:

Faster R3 + Top-P Sampler Replay

This release features major improvements for stable RL training with SkyRL: support for top-p sampler replay, as well as throughput improvements for rollout router replay (R3) for large MoE models.

Top-P Sampler Replay

SkyRL now supports top-p and top-k sampling during RL, using the Keep Sampling Mask approach from DeepSeek-V3.2. With generator.inference_engine.enable_return_sample_support_set=true, vLLM returns the candidates that survived filtering for each generated token. With trainer.algorithm.enable_sample_support_replay=true, the trainer normalizes each token's logprob over that same set. The support travels through the same packed path as the routes, and the trainer scores it without materializing full-vocabulary logits. It works on both Megatron and FSDP.

Bounded sampling helps keep entropy from collapsing over long runs. Example scripts are here.

PRs: #2237, #2238, #2239, #2240, #2241

Rollout Routing Replay (R3)

SkyRL already supported rollout routing replay (R3), but the routing data gets large at scale: one expert ID per token, per MoE layer, per top-k slot. For Nemotron 3 Super (120B), that's nearly 900× the size of the tokens themselves, and the original implementation made each trainer step more than 6× slower.

Most of that cost was CPU time spent serializing, parsing, and copying routes, as well as spillover to disk from the ray object store. In v0.4.0, routes stay in compact packed arrays from vLLM to the trainer, large objects are transferred through ray object store via zero-copy views, and multi-turn generators only send the routes for each new turn. With these improvements, throughput for training Nemotron 3 Super with R3 goes from 111 to 483 tokens/s/GPU.

This release also fixes several multi-turn correctness bugs, including one where replay could silently use stale routes. To enable R3 (Megatron only), set generator.inference_engine.enable_return_routed_experts=true and trainer.policy.megatron_config.moe_enable_routing_replay=true.

PRs: #2232, #2233, #2234, #2235, #2236

Full FP8 + MXFP8 Training

SkyRL now supports Blockwise FP8 and MXFP8 accelerated training and rollout with on-policy weight sync. In long-run RL experiments, the FP8 configurations closely track BF16 convergence while reducing end-to-end step time by up to 23%. More details can be found in our blog post.

PRs:

  • Full FP8 weight sync: #1898
  • MXFP8 weight sync: #2072

Weight Syncing

Weight sync was reworked substantially in v0.4.0. SkyRL now uses vLLM’s new abstractions that decouple trainer send logic and inference receive logic. The result is an easier path for customizing weight transfer for RL frameworks built on top of vLLM.

Delta Weight Syncing

RL training on disaggregated clusters allows users to better utilize available resources in disparate regions , different cloud providers or different hardware. A key bottleneck for disaggregated RL is weight transfer between inference and trainer nodes. During RL training, most of the weights are typically not updated in a given training step (only ~1-3% are updated). Delta weight sync enables efficient disaggregated RL over ethernet. The trainer computes byte-exact XOR deltas against the previously published BF16 checkpoint-format weights, compresses the deltas, publishes a manifest, and asks inference workers to fetch/process the next version before pausing generation. During the weight update, inference workers load the prepared weights from disk.

The implementation supports disk, S3, and GCS-style transfer paths and overlaps receiver-side fetch with generation to reduce idle time in fully async RL setups.

PR: Add Delta Weight Sync Support in SkyRL

Sharded RDT (NIXL)

SkyRL now includes an efficient sharded weight transfer backend that leverages NIXL via Ray Direct Transport (RDT). This is useful for very large MoE runs where simple NCCL broadcast-based transfer becomes a bottleneck. With NCCL, trainer rank 0 forms a collective communication group with all the inference ranks and transfers full weights via broadcast. With the sharded weight transfer engine, we utilize all trainer ranks in the transfer and only send the shard that is needed. The transfer is further optimized to avoid gathering weights across PP ranks, and skips gathering expert layers. The sharded RDT backend is able to achieve weight transfer for the Kimi K2 model in BF16 in 7.53 seconds on 48 8xH100 nodes (32 nodes for the trainer, 16 for inference). We share more details in our blog post.

PR: Sharded RDT weight sync

LoRA In-Memory Weight Syncing

SkyRL now supports in-memory weight syncing with LoRA training. Typically, weight syncing with LoRA in vLLM is implemented with disk based transfer: trainer saves updated LoRA adapters to disk, and the inference ranks re-load the updated adapters into the adapter cache from disk. We implemented simple in-memory weight syncing for LoRA-based training where LoRA weights are transferred via NCCL/CUDA IPC to the inference ranks, which update the active adapter directly on GPU memory. We will be upstreaming our vLLM-side changes in the coming weeks.

PR: In-memory LoRA adapter sync for Megatron

Large Scale SFT

The native SFTTrainer now supports supervised finetraining on very large (10B-1T+ tokens) datasets. We support loading directly from pretokenized datasets on disk for users who run large-scale, complex data processing pipelines. Datasets are loaded in a memory-mapped fashion for memory-efficient data loading. In v0.4.0, SFT training also supports custom sampling schemes, and mixing multiple datasets, allowing researchers to easily experiment with different data mixtures.

Tinker Server Improvements

In SkyRL v0.3.0, the Tinker API server struggled to support over 1,024 concurrent sample requests. Every sample went through a small SQLite connection pool shared with training futures and heartbeats, and the pool ran out quickly. In v0.4.0, the same server handles 131,072 concurrent trajectories with 212k-token outputs. The throughput improvements came from some small and some major fixes. The first fix was keeping forwarded sample futures and their results in memory, so sampling never touches SQLite (thanks to Trajectory AI!). We then fixed the API server’s bottlenecks one by one: replacing httpx with aiohttp, keeping vLLM completion response outputs in protobuf, and reducing unwanted logging were some of the improvements implemented.

PRs: [#2097](https://github.c...

Read more

SkyRL: v0.3.0

Choose a tag to compare

@SumanthRH SumanthRH released this 17 Jul 01:39

Highlights

HTTP-based inference

We introduced a new HTTP-based inference architecture in v0.2.0. This simplifies inference engine management, allows us to integrate high performance routers like vllm-router and supports advance serving patterns like prefill-decode disaggregation. The legacy inference codepath has now been removed in v0.3.0.

Improved load balancing

By default, SkyRL uses a fixed session ID per trajectory and maps a given session ID to a given vLLM engine replica with consistent-hashing. This maximizes KV cache reuse for a given trajectory. However, simple session-based routing can be suboptimal for overall generation throughput in async RL. We’ve implemented a custom routing policy with vllm-router called sticky_least_loaded that load balances trajectories across replicas with sticky routing per trajectory. On our benchmarks for fully async RL on a multi-turn agentic task (averaging 40+ turns) with a large MoE model (GLM 4.7), this load balancing improves throughput by upto 1.4x. See #1791 for more details.

Dynamic microbatching

We added support for dynamic microbatching to improve training performance. Dynamic microbatching packs sequences in each worker upto a fixed token count, instead of a fixed number of sequences per forward pass. This balances real token counts across microbatches, improving throughput and memory utilization when sequence lengths are highly variable. Training throughput improvements are dependent on the task as well as the model and cluster shape, but we see gains of 2x or more common math and agentic tasks. See: https://docs.skyrl.ai/docs/api-ref/skyrl/config#attr-max_tokens_per_microbatch for more details

Dynamic sampling

We’ve added support for dynamic sampling in fully async RL with a new trainer.fully_async.sample_full_batch flag. When enabled, SkyRL drops zero variance groups (groups where all the trajectories score the same reward) and continues sampling until the training mini batch is filled. Zero-variance groups can negatively affect learning and destabilize training. While these zero variance groups receive zero advantages with GRPO, they still affect learning by biasing the aggregate mini-batch loss. Dynamic sampling ensures that such groups are filtered out while keeping a consistent batch size during optimization steps. See: https://docs.skyrl.ai/docs/api-ref/skyrl/config#attr-sample_full_batch for details on usage.

Cache salt for stale KV blocks

SkyRL now defaults to adding a cache salt based on the current model version to avoid re-using stale KV blocks, improving stability during async RL. See: https://docs.skyrl.ai/docs/api-ref/skyrl/config#attr-use_cache_salt for usage

(Experimental) Multi-LoRA support with Tinker

We’ve implemented initial support for multi-LoRA training with the Tinker API. Multi-LoRA training enables researchers to share compute across experiments with a warm, multi-tenant Tinker server. Multi-tenant training can reduce GPU memory requirements by sharing a fixed base model with multiple LoRAs on the trainer and inference engine. Since generations are typically long tailed, multi-tenancy also improves GPU utilization by overlapping requests from concurrent experiments and reduces overall job completions compared to sequential training. For 8 concurrent experiments with Qwen3-4B-Instruct-2507 on a single H200 node, multi-LoRA training can provide E2E speedups of 2.8x. Multi-LoRA support in SkyRL has been co-developed with Trajectory AI, who have shared more results in their blog post.

Native SFT support

SkyRL supports a new SFTTrainer for native SFT training, providing researchers a unified post-training library for SFT and RL. Features added:

  • Diverse dataset formats: SFTTrainer supports instruction following and conversational datasets, including support for tool messages.
  • Sequence packing: Sequence packing packs the global training batch into bins to reduce wasted computation on padding tokens, improving throughput in some settings by 2.75x.
  • Custom callbacks and custom sampling support: Custom callbacks and custom sampling support allow users to modify almost any part of the training loop - bringing data mixtures, using curriculum sampling, logging custom metrics, and more.
  • VLM SFT: Initial support for SFT on VLM models with the Megatron backend, contributed by Suhas!

(Experimental) Speculative decoding with MTP

SkyRL now has experimental support for Multi-Token Prediction (MTP): train a model’s native MTP heads and use them during generation to improve inference throughput. MTP training is implemented in the Megatron backend, by adding a draft loss that trains the MTP heads on the next token prediction task in a decoupled way from the policy head. The updated MTP heads are synced along with the rest of the policy model weights during weight transfer. Speculative decoding with MTP improves generation throughput by upto 2x for Qwen/Qwen3.5-9B on the DAPO recipe (synchronous RL) by reducing tail latency for stragglers. Currently, we only support MTP training with the Megatron backend. We are actively working on benchmarking for multi-turn and async RL scenarios. See: https://docs.skyrl.ai/docs/examples/spec_decode for more details

(Experimental) INT4 fake QAT training

We’ve integrated experimental support for INT4 Quantization-Aware training (QAT). INT4 quantization enables serving extremely large models like Kimi K2.6 (1T+ parameters) on a single 8xH200/ 8xB200 node. However, using INT4 precision for rollouts but BF16 for training can increase trainer <> inference mismatch and destabilize training. INT4 “fake” QAT works by emulating INT4 computation during training by discretizing the BF16 weights into INT4 (in floating point) and then restoring them. We’ve added support for INT4 QAT training with LoRA as an initial step in supporting large models like Kimi K2.6. See the configuration guide: https://docs.skyrl.ai/docs/api-ref/skyrl/config#attr-fake_int4_qat for more details. Thanks to Casper for the contribution, and to Miles for the original work!

What's changed

  • [chore] Move flash attention wheel specification to uv.sources by @SumanthRH in #1546
  • [tinker] Fix optimizer offload in SkyRL train backend after weight sync by @SumanthRH in #1550
  • [bug][tinker] Fix vLLMRouter init in SkyRLTrainBackend by @SumanthRH in #1551
  • [bug] Fix failing CPU tests after #1544 by @SumanthRH in #1555
  • [train] Support packing for CUDA IPC transfer with new inference codepath by @SumanthRH in #1557
  • support trajectory-based agents and OpenHands (#1184) by @arteemg in #1548
  • Update skyrl and skyrl-gym versions after 0.2.0 release by @SumanthRH in #1562
  • [chore] Redirect output to infrastructure logs for VLLMServerActor by @SumanthRH in #1563
  • [harbor][step-wise] Make Harbor use step-wise training by @CharlieFRuan in #1542
  • [docs][example] LoRA config for Geometry3k by @nithinvc in #1569
  • [tinker] Fix single request batching in TinkerEngine by @pcmoritz in #1489
  • [skyrl][tinker] Multi-modal Tinker Sampling by @nithinvc in #1484
  • [feat][megatron] Add freeze_moe_router config to skip router parameters by @SumanthRH in #1570
  • [bug] Fix megatron LoRA with new inference codepath by @SumanthRH in #1572
  • [tinker] Make it clear that a snapshot with persist=false is a marker by @pcmoritz in #1565
  • [tinker] Support PPO loss with Tinker and add critic model in SkyRLTrainBackend by @tamoghnokandar in #1389
  • [train] LoRA-only weight syncing for Megatron by @hao-aaron in #1360
  • added log_example_interval by @agolajko in #1522
  • [ci] Add CI for supported models for megatron backend, and add supported model page to docs by @erictang000 in #1508
  • [tinker] add /api/v1/client/config POST endpoint by @erictang000 in #1578
  • [ci] skip megatron deps for non-megatron CI, disable sample packing for MLA on L4 by @erictang000 in #1585
  • [qoc][trainer] Use WorkerDispatch for weight sync instead of policy_model by @CharlieFRuan in #1584
  • [deps] Bump megatron-bridge to latest by @erictang000 in #1586
  • [bug][tinker] Fix data_parallel_size > 1 in SkyRLTrainBackend by @SumanthRH in #1580
  • [fix][train] Fix weight metadata handling for Megatron backend with native vLLM weight sync by @SumanthRH in #1581
  • [CI] Reorg Megatron metadata consistency test by @SumanthRH in #1587
  • [qoc][trainer] Remove unused PPORayActorGroup and Dispatch methods by @CharlieFRuan in #1588
  • [harbor] Use old inference codep...
Read more

SkyRL: v0.2.0

Choose a tag to compare

@SumanthRH SumanthRH released this 23 Apr 00:11

Highlights

VLM Support: SkyRL now supports VLM training, both through the Tinker API as well as via the python entrypoint. We've validated stable training for single and multi-turn datasets with both text-only environments and multi-modal environment outputs. Get started here: https://docs.skyrl.ai/docs/tutorials/vision_language_rl

New inference refactor centralizing on HTTP: We've implemented a new HTTP-based refactor (*in inference_servers) for inference with vLLM. This standardizes all inference interactions over HTTP, further integrating vllm-router as a high-performance router for generation requests. We also support prefill-decode disaggregation, allows users to squeeze out more performance in multi-turn async RL use-cases. The new inference codepath is now the default - to use the legacy (inference_engines/), set _SKYRL_USE_NEW_INFERENCE=0. The inference_engines/ codepath will be removed in the next release.

vLLM Native weight syncing API integration: SkyRL's new inference servers implementation uses vLLM's native weight syncing APIs: https://docs.vllm.ai/en/latest/training/weight_transfer/

Step-wise training improvements: We've made a number of fixes to the stepwise training implementation, addressing correctness issues (#1492), implementing support for fully async training (#1536) and adding prefix-aware merging to avoid redundant forward passes (#1532)

R3 support: SkyRL now supports R3 for stabilizing training with MoE models. Currently this is limited to cases where vllm engine is within a node due to limitations from vLLM.

Nemotron 3 and Qwen 3.5 support: SkyRL now supports Nemotron 3 and Qwen 3.5 models. Nemotron 3 is supported in the FSDP and Megatron backends, while Qwen 3.5 is supported in FSDP, Megatron and Jax backends.

What's Changed

  • [trivial] Fix comments numbering and make some code more concise in trainer.py by @CharlieFRuan in #1283
  • [train][1/N] Native Weight Sync API: NCCL by @hao-aaron in #1271
  • [ci] Fix CI from broken test imports from #1271 by @erictang000 in #1290
  • [fix] Fix cuda ipc weight sync after #1271 by @erictang000 in #1292
  • fix paths for instruction comments to match current location by @linde in #1294
  • [CI] Skip FlashRL integration test in CI and fix failing generation test for new inference codepath by @SumanthRH in #1301
  • Skip output_router_logits for granitemoehybrid models by @eltonjohnfanboy in #1295
  • WIP: Restore PR changes lost during skyrl-train deprecation by @tyler-griggs in #1310
  • [chore] Update skyrl and skyrl-gym versions after 0.1.0 release by @SumanthRH in #1312
  • [lint] Add isort to pre-commit by @SumanthRH in #1267
  • [examples][bug] fix silent eval max generate length not overriding by @erictang000 in #1317
  • [docs] Add explicit eval_sampling_params.max_generate_length by @SumanthRH in #1318
  • [vllm] enable mp distributed executor backend (no multi-node engines) by @erictang000 in #1300
  • [train][2/N] Native Weight Syncing APIs: IPC by @hao-aaron in #1291
  • [algorithm][generator] change overlong filtering to use stop reasons over checking eos token by @erictang000 in #1319
  • Add rollout_is policy loss by @SamuelGabriel in #1314
  • [AsyncRL] Use keep mode for pause and resume by @hao-aaron in #1179
  • [skyrl][inference] Fix port collision when ports are allocated. by @nithinvc in #1302
  • R3 PR: Rollout Routing Replay by @erictang000 in #1273
  • [megatron] enable bucketed weight sync for non-colocated nccl weight sync in megatron by @erictang000 in #1324
  • [fix] Fix placement group bundle ordering for inference engines by @SumanthRH in #1308
  • [train][fix] Fix concurrency limitations in the new inference codepath by @SumanthRH in #1320
  • [megatron][lora] Fix megatron lora weight syncing not initializing buckets correctly by @erictang000 in #1330
  • [train] Add worker_process_setup_hook to set mp start method to spawn by @SumanthRH in #1333
  • [CI] Fix test_inference_engines_generation after vllm 0.16.0 upgrade; Use the correct GSM8k path for test_generator_multi_turn_gsm8k_router_replay by @SumanthRH in #1339
  • [train] Make TrainingInputBatch to PAD only to left, hence response tensors be right-aligned by @CharlieFRuan in #1285
  • Revert "[train] Add worker_process_setup_hook to set mp start method to spawn" by @SumanthRH in #1344
  • [Docs] Add docs on agent integration and step-wise training by @CharlieFRuan in #1347
  • [Docs] Small update on docs by @CharlieFRuan in #1348
  • [train] Add validation for step-wise GeneratorOutput by @CharlieFRuan in #1281
  • [megatron] rebuild weight conversion tasks per sync to prevent stale PP-collective caches with bucketing by @erictang000 in #1345
  • [StepWise] Trivial fix to avg_response_length metric by @CharlieFRuan in #1351
  • [CI] Make MultiItemDataset a global variable after switch to spawn by @SumanthRH in #1346
  • [train] Add support for LoRA in the new inference codepath by @SumanthRH in #1329
  • [bug][algorithm] remove incorrect torch.no_grad() for kl in loss (use_kl_loss=True) by @erictang000 in #1353
  • [transformers] set return dict false for transformers v5 compatibility by @erictang000 in #1325
  • [skyrl][tx] Move ModelInput token extraction to backends by @nithinvc in #1352
  • [tx] Fuse the projection matrices for Qwen3 by @pcmoritz in #1341
  • [tx] Fuse the projection matrices for Qwen 3.5 by @pcmoritz in #1362
  • Add CodeScout project to README by @CharlieFRuan in #1364
  • [train] Patch vLLM v0.16.0 sleep mode to properly free model weights by @CharlieFRuan in #1365
  • [tx] Optimize the decode performance by @pcmoritz in #1363
  • [skyrl] Add ImageChunk and ImageAssetPointerChunk types by @nithinvc in #1361
  • [train] Enable support for the mp backend with the new inference codepath by @SumanthRH in #1355
  • Fix loss_fn_outputs right-aligned slicing in Tinker API path by @CharlieFRuan in #1367
  • [bug] Move server creation and server start in the same thread by @hao-aaron in #1375
  • [router replay] downcast expert router indices to uint8/int16 to reduce space by @erictang000 in #1378
  • [train] Fix double-serialization of TensorBatch in pickle by @erictang000 in #1379
  • Bump vLLM to 0.18 by @hao-aaron in #1374
  • [train] Use a shared semaphore for all generate requests with RemoteInferenceClient; Move tokenization to client by @SumanthRH in #1381
  • [async] Add search r1 fully async script by @CharlieFRuan in #1386
  • [train] Add vLLMRouter in new inference codepath by @SumanthRH in #1385
  • [trainer] refactor dispatch_from_staged to individually serialize DP chunks to avoid materializing whole batch on all workers by @erictang000 in #1376
  • [SkyRL] Introduce /render endpoint to the new http inference client by @nithinvc in #1373
  • [async] Add DAPO fully async script by @CharlieFRuan in #1390
  • [examples] update command paths to include train/ in examples by @erictang000 in #1395
  • [bug] fix sleep bugs by @hao-aaron in #1383
  • [train][2/N] Support for Megatron PP + CP for R3 by @devpatelio in #1335
  • [train] Multi-modal inputs support in FSDP2 by @nithinvc in #1331
  • [bug] Fix weight sync with DP > 1 in non-colocated setups by @SumanthRH in #1399
  • [train] Make vLLMRouter...
Read more

SkyRL: v0.1.0

Choose a tag to compare

@SumanthRH SumanthRH released this 11 Mar 05:04

Highlights

New unified package : This is the first release of the unified skyrl package combining the skyrl-train and skyrl-tx packages. The unified package brings together the FSDP, Megatron and Jax backend under the Tinker API, while still retaining user-facing "frontend" interfaces (Ex: BasePPOTrainer, SkyRLGymGenerator, etc) from the skyrl-train and skyrl-tx packages. For details about the migration, please refer to #1145.

Improved API documentation: We've revamped the API documentation pages for SkyRL The new API documentation pages can be found here: https://docs.skyrl.ai/api-ref

Pythonic Configs: The skyrl-train backend has now fully migrated to using pythonic dataclasses, replacing the older YAML based interface. The configuration hierarchy has also been updated, and the CLI no longer relies on Hydra. Please refer to the documentation for the new configuration hierarchy: https://docs.skyrl.ai/docs/api-ref/skyrl/config

SGLang is no longer supported: SkyRL no longer supports the SGLang inference engine, unifying on vLLM.

vllm 0.16.0 upgrade: This release updates vLLM to 0.16.0

Qwen 3.5 experimental support: SkyRL now has experimental support for Qwen 3.5 models. This is currently limited to the Jax backend.

What's Changed

Read more

SkyRL-Train: v0.4.0

Choose a tag to compare

@tyler-griggs tyler-griggs released this 13 Feb 17:48
332c7cb

Highlights

Tinker API Integration: SkyRL now fully implements the Tinker API, a simple training and sampling API introduced by Thinking Machines Lab. Any training script written against the Tinker API can run locally on your own GPUs using SkyRL's backends with zero code changes. See the Tinker API docs to get started.

Supported Tinker features include:

  • Supervised fine-tuning (cross_entropy loss) and RL training (importance_sampling loss)
  • LoRA and full-parameter fine-tuning
  • Sampling with logprobs via colocated vLLM inference engines
  • FSDP2 and Megatron training backends
  • Lazy inference engine initialization for SFT-only workloads
  • Ephemeral and persistent weight sync modes

Repo Reorganization: The skyrl-tx and skyrl-train packages are being unified into a single skyrl/ folder. The existing packages remain fully functional and will be migrated to new paths shortly.

Megatron Backend for Tinker: The Megatron strategy is now fully supported for Tinker workloads, including RL training with loss_fn_outputs passthrough.

HTTP Inference Integration: A new HTTP-based inference server integration (feature-flagged) enables decoupled inference engine deployments.

Pythonic Configs: Introduced configuration dataclasses as an alternative to YAML-only configuration, with migration of tests to the new system.

Off-Policy Correction Refactor: Refactored truncated importance sampling (TIS) into a more comprehensive off-policy correction config with support for token-level and sequence-level ratio types.

Harbor Integration: Upstream Harbor integration for evaluation, with Modal support and configurable rate limiting.

Documentation: Migrated documentation to fumadocs, with comprehensive Tinker API docs including quickstart, architecture, cookbook scripts, and configuration pages.

New Model Support (TX):

  • DeepSeekV3 implementation with expert parallelism
  • GLM-4.7 Flash support
  • Qwen3 stacked weights optimization

What's Changed

  • [tx] Add experimental SkyRL-train backend that supports SFT by @pcmoritz in #871
  • Add sampling support for Tinker SkyRL backend by @pcmoritz in #999
  • Add checkpointing support for Tinker SkyRL backend by @pcmoritz in #992
  • Unify Megatron and FSDP training interfaces with forward_backward + optim_step by @pcmoritz in #901
  • Implement forward-only pass and populate metrics by @tyler-griggs in #1046
  • Emit loss_fn_outputs with logprobs for RL losses in forward_backward by @tyler-griggs in #1047
  • [tx] Lazy inference engine initialization by @tyler-griggs in #1069
  • Support colocate_all=False in Tinker backend by @tyler-griggs in #1097
  • [skyrl-train] Return loss_fn_outputs for megatron backend to support tinker RL by @erictang000 in #1102
  • [tx][megatron] making megatron skyrl-train worker usable as TX backend by @erictang000 in #1067
  • [tx][train][merge] make the skyrl folder standalone by @erictang000 in #1084
  • [WIP][skyrl] Create new skyrl folder combining tx + train by @erictang000 in #1068
  • [skyrl-train] Add SFT support via forward_backward(loss_fn="cross_entropy") by @pcmoritz in #961
  • Add set_lr() for dynamic learning rate updates from Tinker by @pcmoritz in #978
  • Fix placement group creation in SkyRL-Train backend by @pcmoritz in #1010
  • [skyrl-train][inference] HTTP Inference Integration (Feature-Flagged) 4/N by @CharlieFRuan in #931
  • [skyrl-train][inference] Inference Server Refactor (1/N) by @CharlieFRuan in #899
  • [skyrl-train][refactor] Inference Server Refactor -- RemoteInferenceClient 2/N by @CharlieFRuan in #904
  • [train] Pythonic Configs 1/N - Introduce configuration dataclasses by @CharlieFRuan in #1001
  • [skyrl-train] Refactor TIS to use more comprehensive off policy correction config by @erictang000 in #849
  • [train][Harbor][1/N] Upstream Harbor integration by @CharlieFRuan in #923
  • [Harbor] Add Modal support and bump Harbor version by @CharlieFRuan in #1022
  • [Harbor] Add rate limit for trials/sec and max concurrency by @CharlieFRuan in #1074
  • [tx] DeepseekV3 implementation by @pcmoritz in #889
  • [tx] Add support for GLM-4.7 Flash by @pcmoritz in #1023
  • [tx] Stack weights — Qwen3 by @pcmoritz in #1079
  • [tx] Add EP axis to deepseek by @pcmoritz in #993
  • [tx] chunked logprobs computation for memory efficiency by @pcmoritz in #902
  • [skyrl-train] Add example for 235B LoRA training with Megatron on 4 H100 nodes by @erictang000 in #1000
  • [train] Enable RayPrometheusStatLogger for async vLLM engine by @CharlieFRuan in #900
  • [train][OpenAI] Add generator.served_model_name for /chat/completions by @CharlieFRuan in #970
  • [train] Enable custom chat template for get_response_ids_and_loss_mask_from_messages by @CharlieFRuan in #981
  • [train][vllm] Add enable_log_requests and max_log_len support by @tyler-griggs in #1071
  • [tx] Use WAL mode for sqlite by @pcmoritz in #1054
  • Increase busy timeout for sqlite to avoid database is locked error by @pcmoritz in #1105
  • [tx] Gracefully handle stale save_weights_for_sampler requests on engine restart by @pcmoritz in #1073
  • Migrate documentation to fumadocs by @tyler-griggs in #941
  • Add Tinker integration documentation by @tyler-griggs in #1050
  • [agent] Add YouCom search engine by @caoshiyi in #803

Full Changelog: skyrl_train-v0.3.0...skyrl_train-v0.4.0

SkyRL-Train: v0.3.0

Choose a tag to compare

@SumanthRH SumanthRH released this 03 Dec 17:07

Highlights

Asynchronous training: We now support fully asynchronous training in SkyRL, enabling higher throughput for agentic RL: https://skyrl.readthedocs.io/en/latest/tutorials/fully_async.html

Dependency Upgrades:

  • Upgraded vLLM to 0.11.0, Ray to 2.51.1
  • Megatron: Migrated from mbridge to the newer Megatron-Bridge library. The latter is expected to have more active development and support from NVIDIA.

The updated installation instructions can be found here.

Recipes: We've consolidated a list of end-to-end recipes with SkyRL here for reference runs on math, Text2SQL and search tasks.

SkyRL on Managed Platforms: Guides for running SkyRL on managed platforms such as Anyscale, Runpod and SkyPilot can be found here.

Miscellaneous: Support for GPT-OSS, integration with Pytorch's OpenEnv, support for IPv6 clusters, and more!

What's Changed

Read more

SkyRL-Train: v0.2.0

Choose a tag to compare

@tyler-griggs tyler-griggs released this 13 Oct 18:12
1ed499c

Highlights

This release contains 163 commits from 22 contributors, including 11 new contributors!

Megatron Backend: SkyRL now has full support for the Megatron training backend with 5D parallelism and strong support for large-scale MoE training. Learn more in our Megatron guide and examples.

LoRA Support: SkyRL now supports LoRA training with the FSDP backend and vLLM inference engine. Learn more in our LoRA guide and examples. We will continue aggressively improving LoRA support and performance, tracked in #449.

OpenAI API Compatibility: SkyRL has standardized around the OpenAI API for inference. This means that agents and agent scaffolds can call into the inference engine over the OpenAI API. SkyRL manages the inference engines and will provide a base_url to hit an OpenAI API compatible endpoint.

Integrations: Building on top of our standardization on OpenAI APIs, we integrated several popular environment and agentic projects. A couple highlights include:

What's Changed

Read more

SkyRL-Train: v0.1.0

Choose a tag to compare

@SumanthRH SumanthRH released this 20 Aug 01:20
6c50026

What's Changed

Read more