Skip to content

v0.10.0

Latest
Latest

Choose a tag to compare

View all tags
@samsja samsja released this 13 Oct 22:20
· 0 commits to bbbe9cb1bf0345f5673e12d2c7514050997f2c95 since this release
v0.10.0
9e15490
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

279 commits from 25 contributors (11 new) since v0.9.0.

Highlights

1. Trainer Performance

One MoE runtime and checkpoint layout now serves every model. Runtime gate/up and QKV fusions are on by default, a SonicMoE BF16 backend is available, and MoE compute can be chosen per layer. MoE blocks now compile with fullgraph, FA4 runs through a custom op so compile can trace attention, and the loss is chunked torchtitan-style. CPU optimizer offload reuses pinned buffers, which cut a DeepSeek V4 Flash SFT step from 30.9 s to 6.5 s.

SFT max trainer perf: prime-rl vs torchtitan. Qwen3-235B-A22B, measured back to back on the same nodes (2026-10-02):

run s/step tokens/s/GPU MFU peak memory
torchtitan (9e159aed7) 5.04 3253 40.7% 144 GiB
prime-rl 4.39 3732 46.6% 141 GiB

prime-rl is 1.15× faster with an identical setup: 4 × 8 B300, seq 16k, FSDP + EP 8, full AC, fp32 AdamW, balanced routing. The trainer defaults favor memory (optimizer and activation offloading, compile without fullgraph). For max throughput, keep everything on GPU and compile with fullgraph, as described in Scaling § Max Trainer Throughput Recipe. Reproduce with uv run sft @ examples/advanced/qwen3-235b-a22b/sft-max-perf.toml.

#3411 — Unified MoE runtime and checkpoint layout
#3451 — Runtime gate/up and QKV fusions
#3535, #3518 — SonicMoE backend and per-layer MoE compute
#3753 — Compiled MoE blocks with fullgraph
#3754 — FA4 through a custom op for compile
#3755 — Attention FLOPs with the real head dim, per-step SFT MFU
#3673 — torchtitan-style chunked loss
#3552 — Pinned buffers for CPU optimizer offload
#3419 — Operator-based activation checkpointing
#3917 — Background micro-batch preparation
#4046 — Max trainer throughput recipe and Qwen3-235B example
#4053 — Release the allocator cache after every checkpoint load, so resumed runs without offload no longer peak above a fresh run

GLM-5

  1. cuDNN + FlashMLA sparse attention (#3829): new model.dsa_backend = "cudnn_flashmla" runs the DSA sparse attention forward with FlashMLA (1.9–3.6× faster than TileLang on H200) and the backward with cuDNN (3–4.7× faster). tilelang stays the default.
  2. 12-node 16k SFT recipe (#3829): the GLM-5.3 SFT example uses cudnn_flashmla, and a new 16k-12-node.toml overlay runs 12 nodes at 16k with no CP and no offloading, plus fullgraph and full AC.
  3. RL example on the new kernels (#3968): the GLM-5.3 SWE RL example (131k seq, CP 4, EP 8, FP8 experts) switches to cudnn_flashmla. Entropy and mismatch KL match TileLang to 4 decimals.
  4. GLM-5.3 examples (#3666, #3607, #3605): SFT and SWE RL examples, BF16 serving with vLLM-side FP8, and the GLM-5.3 renderer.

Full GLM-5.3 (78 layers) SFT, H200, seq 16k, fake data:

setup nodes tokens/s/GPU MFU
before: CP 2 + offloading, tilelang 8 465 22.3%
16k-12-node.toml, tilelang 12 701 33.6%
after: 16k-12-node.toml, cudnn_flashmla 12 835 40.1%

On real weights and data (INTELLECT-3-SFT-10K math, 12 nodes), a step drops from 31.2 s to 24.2 s (25.2% → 32.5% MFU). In the RL setting, a step drops from about 16 s to about 12.5 s (8-layer proxy on 1 × 8 H200), which extrapolates to about 22% faster steps for the full model.

DeepSeek V4 Flash

  1. Sparse attention kernel (#3479, #3644): a TileLang flash-style kernel with a dynamic slot count, per-query tile skipping, and no recompiles. On real data a step goes from 19.2 s to 17.8 s (46.7% → 53.6% MFU).
  2. Lightning Indexer (#3504, #3643): scored with the fused FP8 kernel, with autotuning bucketed so packed lengths no longer stall other ranks, and the CP KV gather overlapped with compute: 22.7 s → 19.0 s per step (40.2% → 47.8% MFU).
  3. Fused mHC chain (#3558): the Sinkhorn loop and write-back run as two Triton kernels, and the RMS norms go through quack.
  4. Fused RoPE (#3645): one Triton kernel for all five RoPE sites, bit for bit with vLLM, makes the SFT step 13–14% faster on B300.
  5. cuDNN + FlashMLA sparse attention (#4078): model.dsa_backend = "cudnn_flashmla" runs the same FlashMLA sparse prefill forward vLLM uses for DeepSeek V4, with the cuDNN DSA backward. tilelang stays the default.
  6. Trainer matches vLLM's FP8 numerics (#3956, #3955, #3647): the trainer reads the same FP8 KV cache as vLLM bit for bit, runs the grouped attention output projection (wo_a) in FP8 like vLLM, and casts FP8 block weights exactly as vLLM's online quantizer does. Removing the DeepGEMM MoE wrapper overhead also makes FP8 MoE training faster than bf16 on H200.

Qwen3.8 Flash Next n-gram embeddings

The n-gram (engram) embedding tables are sharded by vocabulary rows across data-parallel ranks, independently of EP, instead of being replicated on every GPU (#3434).

2. RL Algorithms

IPO (INTELLECT Policy Optimization) is the new default loss (eps = 0.3, no KL term). In async RL the trainer and inference engine never agree exactly on a token's probability. IcePop masks a token when the ratio $\pi_{\text{train}}/\pi_{\text{infer}}$ leaves a fixed band. IPO masks it when the probability itself moved too much:

$$\left|\pi_{\text{train}}(y_t) - \pi_{\text{infer}}(y_t)\right| > \epsilon$$

A ratio misjudges both ends: a rare token going from 0.01 to 0.06 (6×) gets masked, while a likely token going from 0.90 to 0.20 (0.22×) is kept. The IPO mask is double-sided, ignores the advantage sign, and has one knob.

On agentic SWE RL with GLM-4.5-Air (no router replay, up to 64 steps off-policy), IcePop collapsed around step 1.2k when the trainer–inference KL blew up more than 20×. IPO at eps 0.2 and 0.3 trained through it. eps = 0.3 masks only about 0.25% of tokens, compared with 2% for the old 0.1.

IPO vs IcePop on GLM-4.5-Air SWE RL

Full setup, how eps was chosen, and configs to reproduce are in docs/ipo.md (#4049).

IcePop, PPO, and CISPO remain supported as opt-in losses ([trainer.loss] type = ...). GRPO gains a length-weighted baseline and exact prompt-mean loss aggregation. Top-p/top-k sampling is replayed exactly in the trainer, the trainer batch stays a constant size by backfilling filtered samples, and RL freezes the MoE router by default.

#3418, #3586, #4049, #4052 — IPO as the default loss, and its docs
#3526, #3697 — IcePop, PPO, and CISPO losses
#3789, #3798 — Length-weighted baseline and prompt-mean aggregation
#3431 — Top-p/top-k sampling replay
#3463 — Constant trainer batch size
#3797 — Frozen MoE router in RL

3. uv run eval

The eval CLI moves from verifiers into prime-rl. One entrypoint evaluates any model on any environment, standalone or as the online evals of a training run. It runs through env servers with adaptive concurrency and the local dashboard, takes multiple [[source]] blocks, and --resume continues an interrupted run from its trace stream. Every train and eval source now has a select block for picking tasks, and source groups share env and select defaults.

#3471 — Consolidated eval CLI
#3413, #3690 — Resumable standalone evals
#3703 — Per-source task selection
#3695, #3714 — Shared group defaults for train and eval sources
#3702 — Timed-out rollouts as their own outcome
#3826 — Resume requires the saved resolved config to match (--resume.skip-checks to override)

4. New Models: DeepSeek V4 Flash, Qwen3.8 Flash Next, Nemotron-H, and Multimodal RL

Every supported architecture now runs on a PrimeRL implementation; the Hugging Face fallback is gone.

  • DeepSeek V4 Flash: custom kernels for sparse attention, the FP8 Lightning Indexer, the mHC chain, and RoPE (see Trainer Performance). Supports ring context parallelism.
  • Qwen3.8 Flash Next (text): engram support (n-gram embeddings sharded by vocabulary rows), indexed sparse attention kernels from prime-kernels, Gated DeltaNet, hyper-connection residuals, and sigmoid-gated MoE.
  • Nemotron-H (Nano and Super): Mamba-2 on the upstream Triton SSD kernels, with state resets at packed-document boundaries and Ulysses context parallelism.
  • Multimodal RL: images now stay inline through rollouts, packing, and transport, instead of being shipped as preprocessed pixel tensors. vLLM returns the expanded prompt IDs and image placeholder ranges, so each image token lines up exactly in training. The trainer builds vision inputs lazily through a per-family adapter (Qwen-VL, Kimi K2.5).

#3325, #3479, #3504, #3508, #3558, #3644, #3645, #4040 — DeepSeek V4 Flash, kernels, CP, and vLLM serving fixes
#3434 — Native Qwen3.8 Flash Next
#3426 — Native Nemotron-H
#3320 — RL from inline multimodal rollouts
#3773 — Remove the HF model fallback

5. Trace Viewer

Every episode is written once to an append-only stream, and the trainer annotates it afterwards. The dashboard overlays per-token entropy, the trainer↔sampler mismatch, and the IPO mask on the loss mask. New Semantic and Replay views show agent and subagent structure and replay a rollout in real time. Traces have persistent URLs, and the overview has a global filter.

#3444, #3465 — Trace annotations, overlays, and chunked streams
#3455 — Semantic agent traces
#3456 — Timed terminal replay
#3470, #3478, #3721 — Task/judge evidence, compactions, and subagent status
#3725, #3759 — Global filter and persistent trace URLs

6. Inference and Weight Transfer

vllm-router now defaults to sticky least-loaded routing, which sends new sessions to the least-busy engine and keeps each session on its engine for prefix-cache hits. NCCL updates always send HF-format weights and let vLLM apply its own FP8 quantization. NIXL transfers are pipelined through ModelExpress, and a Dynamo admin plane is available for weight sync.

#3563 — Sticky least-loaded routing
#3580, #3581 — Online inference quantization
#3340, #3341 — ModelExpress and pipelined NIXL pulls
#3491 — Dynamo admin plane
#3608 — EP weight filter on by default
#3900 — Packed /generate per-token payloads and tuned API-server GC, removing multi-second GC pauses under RL load
#3830 — Router replay with disaggregated P/D on the V2 runner
#4045 — MXFP8 online quantization ([inference.vllm] quantization = "mxfp8")
#4031 — Backport of vLLM's SHM buffer ownership fix, so router replay no longer reads overwritten expert IDs

7. Job Control and Batch Transport

Each component now exposes an HTTP control API, so a running RL or SFT job can be paused at a chosen step, saved, and stopped. Trainer ranks now pull each batch step from the orchestrator, so the trainer only holds the batch it is training on.

#3974 — Pause a running job through a control API on each component
#4065 — Trainer pulls each batch step
#4082 — RL component serialization exposed from the config package

Dependencies

  • torch: 2.9 → 2.13, on CUDA 13. #3425
  • vllm: 0.26 → 0.31.0. #3425, #3430, #3617, #3716, #3888
  • verifiers: 0.3.1 → 0.4.0. #4056
  • renderers: 0.1.10 → 0.1.12, with the multimodal extra. #3431, #4056
  • vllm-router: 0.2.0 → 0.2.2. #3563, #3718
  • prime: 0.6.19 → 0.8.0; the prime CLI replaces the direct prime-sandboxes dependency. #3717
  • prime-runs 0.1.4 is a new dependency for platform runs. #3459, #3928
  • New optional extras: sonic-moe, flash-mla, and dynamo. The gpt-oss extra and mamba-ssm dependency group are removed; mamba-ssm is now part of the gpu extra. #3535, #3829, #3491, #3424, #3856

Breaking

  • Eval entrypoint: uv run evals is now uv run eval. The verifiers in-process CLI is uv run vf-eval. Replace num_examples with select.limit. #3471, #3703
  • Source groups: move orchestrator.group_size and orchestrator.algo into [orchestrator.train]. Group-level env, select, sampling, and group_size are defaults that each source inherits. #3714
  • RL loss: the DPPO default loss is removed and IPO is the default. Rename ipo_threshold to eps. Remove dppo_mask_low, dppo_mask_high, and kl_tau. The DPPO is_masked_low/high and masked_advantage_* metrics are removed; use is_masked. #3418, #3810
  • CPU offload: replace model.fsdp_cpu_offload, model.optim_cpu_offload, and model.full_offload with model.offload = "optimizer" | "full" | "none". The FSDP offload mode, full_offload.numa_bind, and full_offload.cpu_optimizer_backend are removed. #3988, #3781
  • Model implementation: model.impl is removed. Architectures without a PrimeRL implementation fail at setup, and the trainer no longer loads qwen3_vl, qwen3_vl_moe, or gemma4. #3773
  • MoE checkpoints: the runtime and checkpoint layout changed. Reusable <conversion_dir>/prime caches need the .prime-v1 marker, so legacy Prime layouts fail before loading. Reconvert from HF. #3411
  • Batching: orchestrator.token_batch_size is removed; use batch_size. orchestrator.train.filter_zero_advantages is removed, and zero-advantage samples are always filtered. #3775, #3463
  • Weight transfer: weight_broadcast.quantize_in_weight_transfer is removed; set [inference.vllm] quantization = "fp8_per_block" instead. quantization only accepts fp8_per_block, mxfp8, or unset. orchestrator.ckpt.wait_for_weights_timeout is removed; weight_broadcast.timeout (now 3600 s) covers the startup wait. #3581, #3580, #3980
  • Run directory: configs move to configs/attempt_<n>/, with configs/latest pointing at the newest launch. Read configs/latest/resolved/ instead of configs/resolved/. Traces move from rollouts/step_N/... to monitors/file/traces/stream.jsonl plus annotation streams. #3405, #3444
  • Kubernetes: the Helm chart and trainer metrics_server are removed. SLURM is the supported multi-node path. #3767
  • Renamed or removed config fields:
    • tasks_per_minute → dispatch_per_minute. #3991
    • model.index_cache is removed. #3984
    • model.ac_offloading.pin_memory is removed; offloaded activations always use pinned CPU memory. #3972
    • trainer.model.fusions.raise_on_fail is removed. #3778
    • optim.type = "sgd" and ckpt.skip_progress are removed. #3770
    • orchestrator.model.lora, orchestrator.ckpt.keep_last/keep_interval, and dynamo on non-policy clients are removed. #3983
    • Typed [inference.vllm] fields seed, enforce_eager, dtype, trust_remote_code, enable_dbo, max_loras, and all2all_backend now pass through to vLLM untyped, so --no-vllm.enforce-eager becomes --vllm.enforce-eager false. #3986
  • Python API: import pathing helpers from prime_rl.utils.pathing, not prime_rl.utils.utils. prime_rl.orchestrator.patches is removed. #3769
  • /generate responses: non-streaming responses return packed base64 payloads for routed_experts, completion_logprobs, and the CSR sampling_mask, and no longer include top_logprobs. Clients need verifiers >=0.4.0 and renderers >=0.1.12. #3900
  • Installation: uv sync --group mamba-ssm no longer works; use --extra gpu. #3856

Changed defaults

  • IPO uses eps = 0.3 and no KL term. #3586
  • RL freezes the MoE router (freeze_moe_router = "auto"). Set trainer.model.freeze_moe_router = false to train it. A frozen-router checkpoint cannot resume with the router unfrozen. #3797
  • optim.weight_decay = "auto" resolves to 0.0 for RL and 0.01 for SFT. #3815
  • moe_router_dtype = "auto" resolves to fp32 for RL and bf16 for SFT. #3757
  • orchestrator.constant_trainer_batch_size = true backfills filtered samples. #3463
  • vllm-router request timeout rises from 30 minutes to 4 hours. Set router.request_timeout_secs = 1800 for the old cutoff. #3654
  • dist_timeout_seconds drops from 3600 s to 900 s. #4019
  • FSDP reduce-scatter buffer cap is now configurable (model.reduce_scatter_max_input_buffers) and defaults to 1; fp32-router MoE runs previously used 2. #3912
  • inference.vllm.enable_ep_weight_filter = true. #3608
  • DeepSeek V4 trains against vLLM's FP8 KV cache (model.dsv4_kv_precision); set it to "bf16" for the old behavior. #3956
  • RL data.num_workers = 1. #3976

New Contributors

Contributors

@mikasenghaas, @faresobeid, @samsja, @garrett361, @S1ro1, @hallerite, @xeophon, @JannikSt, @cabrinha, @MarioSieg, @kcoopermiller, @nxphi47, @eexwhyzee, @eligotts, @hubert-marek, @biswapanda, @JohannesHa, @parkerpettit, @yafshar, @Alex-Shan, @rasdani, @akirillo, @Apocrathia, @artzhuravel, @skpulipaka26

Contributors

cabrinha, biswapanda, and 23 other contributors
Assets 2
Loading