Skip to content

SkyRL: v0.5.0

Latest

Choose a tag to compare

@SumanthRH SumanthRH released this 17 Nov 08:58

Summary

SkyRL v0.5.0 continues our work enabling large scale RL on diverse models, with the following key updates:

  • Model Support: Support for the following new models:
    • Beam-1.0 (501B-A23B)
    • DeepSeek-V4.1-Flash
    • Qwen3.8-Flash-Next
  • RL stability
    • Score Centering
  • Scalability
    • Optimizer Offload to Disk
    • 1M-token context for GLM-5.3-Flash
  • New native On-Policy Distillation (OPD) trainer
  • Vision Language Models: sequence packing, context parallelism, and images in agentic RL

In addition, this release updates a number of defaults in SkyRL:

  • Score centering is enabled for compatible loss functions
  • DPPO policy loss
  • Fused LM head

Score Centering

SkyRL now supports score centering (Marek and Ryabinin) to keep RL stable when the rollout and training engines disagree. 4-bit inference for routed experts is an easy way to boost decode throughput on Blackwell, but it creates train-inference mismatch (TIM) that router replay and importance sampling alone can't handle, leading to reward collapse. SkyRL computes the score centering correction exactly over the recorded top-k/top-p sampling support, reusing the packed sample-support path from v0.4.0, without materializing full-vocabulary logits. It works on both Megatron and FSDP.

On an antibody design task with GLM-5.3 (NVFP4 W4A4 rollouts, BF16 trainer), the run without score centering collapsed within 20 steps as TIM and entropy blew up, while the run with score centering kept hill climbing for 80 steps with stable TIM. This integration was contributed by Dominic Yurk from Lila Sciences.

Score centering is automatically enabled when sample-support replay and a compatible sampler-anchored loss are configured. To enable it explicitly:

trainer.algorithm.policy_loss_type=rollout_is
trainer.algorithm.enable_sample_support_replay=true
trainer.algorithm.enable_score_centering=true

PRs: #2484, #2608

Optimizer Offload to Disk

For large MoE models like Beam-1.0-501B-A23B, optimizer state takes ~18 bytes per param (~9 TB in total), which can exhaust host RAM even with CPU offloading. SkyRL now supports offloading optimizer state to disk: each rank stores its optimizer state on node-local NVMe and streams it through the GPU one chunk at a time. Training Beam on 8 B200 nodes, host RAM usage drops from ~13.5 TB to <= 3.2 TB, with each optimizer step taking ~25-50s per rank, comparable to CPU offloading (~35s for similar model sizes).

PR: #2490

Long Context: 1M Tokens for GLM-5.3-Flash

Full fine-tuning of GLM-5.3-Flash can now train on up to 1M tokens per sequence, a 64× increase, through work in four areas:

  • DeepSeek Sparse Attention: GLM's DSA layers now actually run on a sparse kernel (TileLang) instead of silently falling back to dense attention. The DSA indexer is chunked and split across TP ranks.
  • Activation recompute and offloading: full activation recompute, releasing MoE router state after forward, and offloading each layer's saved input to pinned CPU memory.
  • Context parallelism: head-wise (Ulysses-style) CP for GLM's KDA layers, all-gathering the small hidden state instead of all-to-all on projected tensors.
  • Avoiding unnecessary precision upcasts in the mHC projection and MoE combine.

More details in our technical blog.

PRs: #2388, #2411, #2412, #2413, #2414, #2418, #2422, #2433, #2434, #2437

New Model Support

Beam

SkyRL now supports training reflection/Beam-1.0-501B-A23B, building on our Day 0 support effort. LoRA, full fine-tuning and MXFP8 RL are supported out of the box. On the DAPO recipe, all three improve AIME25 avg@8 from ~61% to ~79% in 50 training steps. With R3 enabled, the average logprob diff between Megatron and vLLM stays under 0.01 for LoRA and BF16 full fine-tuning, and under 0.03 for MXFP8 RL.

PRs: #2604, #2618, #2619, #2625

DeepSeek-V4.1-Flash

SkyRL now supports RL on DeepSeek-V4.1-Flash (text only) with the Megatron backend and vLLM inference:

  • LoRA RL with rank-32 adapters on attention and expert layers, synced to vLLM every step. Validated end to end on GSM8K with 8×B200 nodes.
  • Full fine-tuning, with trainer weights re-quantized to FP8/MXFP4 and synced in FP8 without a BF16 round trip.
  • Released checkpoint as is: the trainer loads the released FP8/MXFP4 weights directly, and vLLM serves them with LoRA enabled.

PRs: #2551, #2565, #2566, #2567, #2610, #2612, #2646, #2721

Qwen3.8-Flash-Next

SkyRL now supports Qwen3.8-Flash-Next training on the Megatron backend with both LoRA and full finetuning. We worked with NVIDIA to integrate the cuDNN QSA sparse attention kernels, making training 30% faster than our FlexAttention baseline at 30K tokens with much lower memory. The model scales with expert parallelism (which also shards the frozen 51B-parameter n-gram embedding table) and TP with sequence parallelism, and trains on sequences of up to 262K tokens with context parallelism.

PRs: #2513, #2607

On-Policy Distillation (OPD) Trainer

Building on our previous exploration, SkyRL now has an official on-policy distillation entrypoint and trainer. We implement sampled-token OPD: the teacher grades each token the student samples, giving dense per-token signal.

  • Fewer steps than RL: on DAPO-Math-17k, a Qwen3-4B-Base student matches its DAPO-trained teacher's AIME-2024 accuracy in 20 steps, where DAPO took 90.
  • Minimal teacher overhead: the teacher runs on its own vLLM engines and scores each prompt group as soon as it finishes, overlapping generation and scoring.
  • Combine task rewards with teacher-scored advantages via trainer.algorithm.opd.use_task_reward=true.
  • Managed or self-served teachers: use an existing vLLM endpoint, or let SkyRL launch the teacher.

Coming next: more OPD metrics (#2624, #2632), stabilization strategies (#2622), and multi-teacher OPD (#2679).

Vision Language Models

This release improves VLM training on the Megatron backend for Qwen3-VL and Qwen3.5, and adds images to agentic RL:

  • Sequence packing: VLM sequences are now packed without padding, increasing training throughput by 54% on Geometry-3K with Qwen3.5-9B. Set trainer.remove_microbatch_padding=true.
  • Context parallelism: split long sequences across GPUs with trainer.policy.megatron_config.context_parallel_size=<n> (example script).
  • Images in agentic RL: skycap now captures agent rollouts that include images, so the trainer sees the same image features as vLLM. See the new SWE-bench Multimodal recipe.

PRs:

Changelist

  • [chore] Update version after 0.4.0 release by @SumanthRH in #2390
  • [ci skip][docs] add doc to handoff to agent for new model support by @erictang000 in #2391
  • [deps][megatron] Bump transformer-engine to 2.19.0 by @erictang000 in #2387
  • [megatron] GLM-5.3-Flash DSA: run the fused TileLang SparseMLA kernel by @avigyabb in #2388
  • fix: preserve FIFO trajectory rate admission by @SumanthRH in #2383
  • [deps][megatron] Add flash-mla from skyrl-wheels for the cudnn DSA kernel backend by @erictang000 in #2392
  • [r3] Allow router replay on the vLLM ray executor; refuse inference pipeline parallelism by @erictang000 in #2332
  • [ci skip][ci] Raise H100-GPU-CI Anyscale job timeout to 12000s by @erictang000 in #2397
  • [megatron] Opt-in hybrid DSA kernels: TileLang indexer top-k with cuDNN/FlashMLA sparse attention by @erictang000 in #2395
  • [vllm] Keep R3 routed-experts capture bound across MoE kernel rebuilds (vllm#59455 backport) by @erictang000 in #2396
  • [fp8] GLM-5 / GLM-5.3 FP8 rollout support: glm_moe_dsa ModelFp8Spec + per-block FP8 param fix by @erictang000 in #2398
  • [metrics 1/5] Add scoped vLLM offload and latency window metrics by @SumanthRH in #2364
  • [metrics 2/5] Report aggregate engine throughput imbalance by @SumanthRH in #2366
  • [metrics 3/5] Publish weighted fresh-run vLLM summaries by @SumanthRH in #2367
  • [metrics 4/5] Publish Grafana run-name annotations from the Ray head by @SumanthRH in #2368
  • [sft] GLM-5.3-Flash full fine-tuning in SFT (Megatron), random-init benchmarking by @avigyabb in #2410
  • [megatron] Random init: drop the TP-replicated parameter broadcast by @avigyabb in #2421
  • [harbor] Per-step trial phase and retry metrics for both Harbor generators by @kouroshHakha in #2342
  • [skycap] Token mode matches on rendered fields, and the renderer decides respellings by @kouroshHakha in #2352
  • [skycap] One MatchKey per call instead of a chain of match-hash helpers by @kouroshHakha in #2426
  • [skycap] Choose which captured paths train: all, final, or a user-defined rule by @CharlieFRuan in #2385
  • [skycap] Expose a server's harness routes to agents in remote sandboxes (quick tunnel, external_host), and use it for Harbor's installed agents by @kouroshHakha in #2429
  • [train] Raise generator failures in async trainers instead of hard-exiting by @dyurk-lila in #2379
  • [skycap] Per-trajectory API keys on the harness routes, required by default when Harbor exposes skycap by @kouroshHakha in #2427
  • [inference] Make inference-server HTTP errors picklable; keep cleanup from masking the primary error by @dyurk-lila in #2380
  • [skycap] Fix format.md where it disagrees with the writer by @jadenfix in #2425
  • [skycap] The exposure's stop grace and cloudflared download timeouts are environment variables by @kouroshHakha in #2435
  • [metrics 5/5] Scope prefill/decode step metrics and run summaries by @SumanthRH in #2401
  • [train] Add optional step and weight-sync timeouts that raise typed errors by @dyurk-lila in #2381
  • [metrics 6/6] Collect vLLM metrics from external SkyRL servers by @SumanthRH in #2420
  • [megatron] MoE: release the token dispatcher's router probs after each MoE forward by @avigyabb in #2433
  • [skycap] Mirror records to remote storage, and say where each is at finish by @kouroshHakha in #2350
  • [skycap] Land the W&B record index and pull on main (#2351, #2436); pull writes a directory per phase by @kouroshHakha in #2441
  • [megatron] mHC: save the bf16 input of the FP32 projection, not its upcast by @avigyabb in #2434
  • [megatron] GLM-5.3-Flash DSA: score and select k-pool indexer pools in query chunks by @avigyabb in #2413
  • [megatron] GLM-5.3-Flash DSA: shard the k-pool indexer's query rows across TP (opt-in) by @avigyabb in #2422
  • [Fix][fsdp] Load a LoRA policy's frozen base in bf16 under trainer.bf16 by @gc-anyscale in #2323
  • [Fix][inference] Always launch vLLM with logprobs_mode=processed_logprobs by @SumanthRH in #2456
  • Return sample sequence IDs from asample by @j316chuck in #2353
  • fix: return sequence IDs for Tinker async sampling by @kalectory in #2394
  • fix: preserve token order in Megatron router replay by @hershg in #2358
  • Fix Tinker custom-loss compatibility with loss:sum metric by @j316chuck in #2455
  • [tinker] Support tinker SDK 0.27 (pin <0.28, loss_fn_config_v2) by @avigyabb in #2457
  • [Fix][CI] Address GPU E2E timeout and Megatron offload assertion (#2458) by @SumanthRH in #2458
  • [examples][dapo] AIME-2025 eval set and an AIME eval-only example script by @SumanthRH in #2466
  • [skycap][bugfix] Give the parser the prompt, so a prefilled <think> stays out of content by @sunnyysetia in #2453
  • [megatron][vlm] Allow sequence packing for VLMs incl. Qwen3.5-VL (CP=1) by @xinyuangui2 in #2357
  • [fix] Release the packed CUDA IPC buffer after in-memory LoRA syncs by @SumanthRH in #2488
  • [megatron] mHC layer: honor mlp_chunks_for_training (chunk the MLP/MoE over the sequence) by @avigyabb in #2468
  • [examples] Support per-token rewards in the DAPO trainer with batched=false by @SumanthRH in #2489
  • [feat][lora] Size MoE expert LoRA slots at the expert adapter rank with normalize_moe_lora=true by @SumanthRH in #2469
  • [chore] Bump vLLM to 0.31.0, flashinfer 0.7.0.post1, Megatron-Bridge/Core, cutlass-dsl, xgrammar by @SumanthRH in #2470
  • [r3] Routed-experts capture with vLLM pipeline parallelism (runtime relay patch) by @SumanthRH in #2472
  • [fix] Demote HTTPX request logs in Ray workers by @SumanthRH in #2480
  • [megatron] Add lora_config.sequence_parallel_input_regather for Megatron (default true) by @SumanthRH in #2493
  • [Perf] Free up temporary memory allocated for LoRA tensors for in-memory weight sync by @SumanthRH in #2492
  • [fix][ci] Use generic Tinker unload endpoint in Megatron test by @SumanthRH in #2495
  • [ci] allow GPU suite time for Mooncake test by @SumanthRH in #2496
  • [megatron] Skip the post-backward empty_cache when the optimizer is fully on CPU by @avigyabb in #2500
  • [vllm] Skip detokenization when capturing sample support by @erictang000 in #2499
  • [docs] Day 0 Support for Beam by @SumanthRH in #2502
  • [fix][skycap] Always update _by_match by @xinyuangui2 in #2461
  • [megatron][vlm] Context parallelism for Qwen3-VL / Qwen3.5-VL by @xinyuangui2 in #2404
  • [megatron] Let a LoRA-wrapped output_layer answer for its base linear by @avigyabb in #2505
  • [skycap][bugfix] Preserve extracted arguments without promoting failed parses by @cbanbury-lila in #2497
  • [LoRA] Add the PEFT adapter import library (parsing, scales, fused stacking) by @avigyabb in #2506
  • [ci] Fix routed-experts PP relay test on L4 and raise H100/Megatron GPU CI timeouts by @erictang000 in #2515
  • [inference] /skyrl/v1/generate: accept multimodal features by @xinyuangui2 in #2501
  • Fix AMD Tinker FSDP Docker runtime by @eddierichter-amd in #2520
  • [megatron] Initialize a LoRA policy from a PEFT adapter (lora.init_adapter_path) by @avigyabb in #2508
  • [skyrl-train] Carry sample support as CSR from capture through replay by @dyurk-lila in #2483
  • [CI] Fix routed-experts PP relay (CUDA graphs) and VLM CP=1 OOM on L4 GPU CI by @erictang000 in #2530
  • [SkyCap] Support OpenAI Responses API by @kyuds in #2510
  • [megatron] chunked_mlp: equal MoE call counts per rank, and router replay (R3) per chunk by @xinyuangui2 in #2532
  • [skyrl-train] Score centering over the recorded sample support by @dyurk-lila in #2484
  • [examples][skycap] Carry the sampler's support logprobs through harbor_skycap by @dyurk-lila in #2485
  • fix(rdt): re-apply composed-loader transforms on replay (Mamba2 A = -exp(A_log)) by @dyurk-lila in #2336
  • fix(megatron): gather updated parameters before weight transfer, HF export and checkpoint save by @dyurk-lila in #2337
  • [megatron] Pad unpacked rows to a multiple of mlp_chunks_for_training by @avigyabb in #2540
  • Fix H100 startup timeout and CPU Megatron CI regressions by @SumanthRH in #2548
  • [weight sync] Keep fp32 and integer params in their dtype on the wire by @avigyabb in #2543
  • [vllm] Restore FlashInfer CUTLASS MoE constants after a weight sync by @avigyabb in #2544
  • [chore] Upgrade compute config version for H100 CI by @SumanthRH in #2552
  • [megatron] Initialize CPU-offloaded optimizer state before checkpoint save by @avigyabb in #2542
  • [megatron] Recompute output-discarding checkpoints whose inputs need no grad by @avigyabb in #2541
  • [chore] Bump vLLM to 0.32.0, drop the routed-experts rebind patch by @erictang000 in #2557
  • [vllm] Allocate a pipeline stage's KV cache only for the layers it owns by @avigyabb in #2545
  • [megatron] Fix the distributed optimizer for param groups mixing fp32 and bf16 by @avigyabb in #2539
  • [fix][megatron] Call mark_not_offload in the fused LM head path by @dyurk-lila in #2522
  • [megatron] Load the pinned FlashMLA extension before vLLM's bundled copy by @avigyabb in #2563
  • [vllm] Zero DeepSeek-V4.1's BF16 FlashMLA prefill KV workspace (NaN first tokens) by @avigyabb in #2551
  • [megatron] DeepSeek-V4 hybrid attention: implicit causal mask, compression-ratio alignment, no core_attn recompute by @avigyabb in #2561
  • Fix main GPU CI test failures by @SumanthRH in #2556
  • [feat][trainer] Add support for trainer callbacks to FullyAsyncRayPPOTrainer and SFTTrainer's dummy benchmark loop by @SumanthRH in #2550
  • [skycap] Images in token mode by @xinyuangui2 in #2431
  • [CI] Take Anyscale submission files from the base branch on labelled PRs by @SumanthRH in #2572
  • [ci] SkyRL-JAX-GPU: skip fork PRs on the plain trigger, add run_jax_gpu_ci label run by @ArturNiederfahrenhorst in #2535
  • [ci skip] Update OpenEnv links to openenv/OpenEnv by @sergiopaniego in #2571
  • [deps] install FlashInfer on aarch64 (GB200 / GB300) by @kaixih in #2574
  • [megatron] Give the CPU-offloaded optimizer the rank's share of CPU threads by @avigyabb in #2569
  • [skyrl-train][bugfix] Run the sampler weight broadcast off the event loop by @dyurk-lila in #2524
  • [megatron] GLM-5.3 (glm_moe_dsa) with FP8 TP8 vLLM rollouts: LoRA and full-FT DAPO recipes by @erictang000 in #2347
  • [fix] Fix GPU CI after the vLLM 0.32.0 bump; raise the Megatron GPU CI timeout by @erictang000 in #2582
  • [vllm] Defer DeepSeek-V4's model-level finalize to the end of a weight sync by @avigyabb in #2560
  • [vllm] Serve DeepSeek-V4.1 with vLLM's text-only (LoRA-capable) class by @avigyabb in #2565
  • [vllm] Extend the DeepSeek-V4 weight-sync reload patch to V4.1's text model by @avigyabb in #2566
  • Increase Broadcast Event Loop Block Timeout by @benji-cannot-code in #2586
  • [tinker] Honour train_attn/train_mlp from Tinker's LoraConfig on the skyrl-train backend by @ArturNiederfahrenhorst in #2464
  • [feat][megatron] Support optimizer state offload to disk (optimizer_offload_to_disk) by @SumanthRH in #2490
  • [megatron] DeepSeek-V4.1-Flash (text only) model support by @avigyabb in #2567
  • [megatron][vllm] Qwen3.8-Flash-Next (qwen4_exp) support: Megatron bridge, QSA sparse attention, GSM8K + DAPO recipes by @xinyuangui2 in #2513
  • [harbor_skycap] Images through Harbor, a sandbox concurrency cap and labels, SWE-bench Multimodal recipe by @xinyuangui2 in #2450
  • [tinker] Honour train_attn/train_mlp on the Megatron backend by @ArturNiederfahrenhorst in #2529
  • perf(megatron): autotune fused LM-head Triton kernels by @dyurk-lila in #2373
  • feat(megatron): fuse LM-head entropy for RL by @dyurk-lila in #2374
  • feat(opd): On-Policy Distillation Entrypoint by @kyuds in #2256
  • [megatron] Skip per-pass empty_cache by default under expandable segments by @avigyabb in #2600
  • perf(megatron): skip the frozen LM-head weight gradient in fused logprob backward by @avigyabb in #2598
  • Enable Custom Quantization by @benji-cannot-code in #2399
  • honor vllm's cache reset contract by @coreyhu in #2316
  • [megatron] Bump megatron-bridge to 7171584b and megatron-core to ed9c2c8c; retire landed patches by @erictang000 in #2558
  • [megatron] Kimi K3: 4-layer slice and Megatron <> vLLM roundtrip logprob test by @erictang000 in #2605
  • [megatron] Opt-in head-full DSA sparse attention (Ulysses-style all-to-all over TP) by @avigyabb in #2601
  • [deps] Bump flash-linear-attention to 0.6.0 (B200 GDN race fix) by @xinyuangui2 in #2617
  • [harbor_skycap] Give installed agents OPENAI_BASE_URL, so Codex reaches skycap by @ArturNiederfahrenhorst in #2615
  • [feat][megatron] Log model FLOPs utilization (MFU) for policy training by @SumanthRH in #2614
  • feat(megatron): add block-sparse fused LM-head forward by @dyurk-lila in #2375
  • [deps] Bump vLLM to 0.33.0 (torch held at 2.13): WeightSource contract, NCCL/IPC init, NIXL and connector fixes by @SumanthRH in #2604
  • [weight sync] Keep fp32 and integer parameters in their dtype on the sharded-RDT wire by @SumanthRH in #2618
  • [megatron] Fix OOM on native checkpoint save and resume by @sgabriel92 in #2525
  • [docs] Add a Model Support section with per-family model pages by @erictang000 in #2627
  • [models] Beam (BeamForCausalLM) support: full fine-tuning and LoRA on vLLM 0.33 + upstream Megatron-Bridge by @SumanthRH in #2619
  • [megatron] Qwen3.8-Flash-Next context parallelism up to 262K by @xinyuangui2 in #2607
  • [train] Default to DPPO, prompt_mean loss reduction, and no GRPO std norm by @erictang000 in #2635
  • [train] Default enable_score_centering to auto by @erictang000 in #2608
  • [megatron] Context parallelism for GLM-5.3-Flash (head-wise KDA, all-gather DSA) by @avigyabb in #2418
  • [docs] Fix Vercel MDX parse error in Qwen3.8 model page by @SumanthRH in #2681
  • [megatron] chunked_mlp: router replay under selective "moe" recompute; sync with Megatron-LM#8463 by @xinyuangui2 in #2559
  • [megatron] DeepSeek-V4.1 load: dequantize FP8 weights in the released 32-wide blocks by @avigyabb in #2610
  • [megatron] int64 row offsets for megatron-core's Triton kernels (fused mHC, MLA RoPE) [enablement] by @avigyabb in #2646
  • [harbor] Bump Harbor to v0.14.0; use its Daytona labels instead of LabelledDaytonaEnvironment by @xinyuangui2 in #2695
  • [deps] Move to torch 2.14.0 with uv-native CUDA extension wheels by @SumanthRH in #2625
  • [megatron] DeepSeek-V4.1 full-weight sync: 32-wide FP8 export re-quantization; keep FP8 / e8m0 dtypes on the wire by @avigyabb in #2612
  • [ci] Calibrate GSM8K E2E checks for DPPO defaults by @SumanthRH in #2682
  • [harbor_skycap] Drop the remote-sandbox concurrency-cap check by @CharlieFRuan in #2716
  • [trivial][qoc] Fold _cache_salt_model_name into build_vllm_cache_salt by @CharlieFRuan in #2719
  • [ci] Finish Megatron GPU suites and validate final W&B history by @SumanthRH in #2717
  • [tinker] Reject backend_config values the server overrides on the SkyRL-Train backends by @ArturNiederfahrenhorst in #2516
  • [qwen3.8] Faster LoRA DAPO recipes: one MoE chunk, merged rollout weights by @xinyuangui2 in #2726
  • [vllm] Retire the DP TCPStore port-window workaround now that vllm#50969 is in 0.33.0 by @ArturNiederfahrenhorst in #2729
  • [skyrl-train] Release GPUs when create_model fails part-way by @ArturNiederfahrenhorst in #2555
  • [test] Give broadcast event-loop check a 1s deadline by @SumanthRH in #2727
  • [train] Check Megatron's expert grid at config time; default MEGATRON_EP to 1 in run_codecontests.sh by @ArturNiederfahrenhorst in #2639
  • [harbor_skycap] Stop a run whose first batch has nothing to train by @ArturNiederfahrenhorst in #2629
  • [megatron] DeepSeek-V4.1: accept megatron-core's context-parallel boundary kwargs in CSA2 core attention [fix] by @avigyabb in #2721
  • [docs] Inkling2 model page by @erictang000 in #2731
  • [CI] Shard H100-GPU-CI into models/other; replace 8-GPU QSA rows with cp4_ep4 by @SumanthRH in #2733
  • [CI] Fix H100-GPU-CI on main: FP8 health-wait ceiling, one shard and one run at a time by @SumanthRH in #2743
  • [integration] Move the Harbor and Harbor-skycap generators into skyrl/integration by @xinyuangui2 in #2741
  • [inference] Encode data-plane request bodies with orjson [optimization] by @avigyabb in #2749

New Contributors

Full Changelog: skyrl-v0.4.0...skyrl-v0.5.0