Repository navigation
Highlights
Important
Graduated Distributed RL (tunix.dist_rl)
A production-ready, modular RL framework promoted from tunix/experimental that enables training at the 10k+ accelerator scale, complete with unified launchers, Kubernetes deployment support, and GSM8K, FrozenLake, and DeepSWE recipes (#2999, #3014, #3134, #3175, #3205, #3231).
Zero-TIM Canonical Pallas TPU Kernels
Achieves zero log-prob difference (bitwise-identical logprobs) between the trainer and sampler for the first time on TPU, unlocking truly stable on-policy RL. Read more in our blog post (#2600, #3194, #3302).
Score Centering for Disaggregated GRPO (score_centering)
Eliminates sampler–trainer distribution bias in disaggregated and off-policy GRPO via a zero-value gradient correction over sampler top-k logprobs and the rescaled trainer tail, complete with async rollout top-k collection, clipped token-IS composition, and CLI support. Read more in the paper (#3316, #3409).
Additional Highlights
- Gemma 4 Multimodal & Scale Support: Added Vision (#1601, #1606, #1607), Audio (#1625, #1634), 12B/12B-it (#1570), and 26B/31B (#2875) support,
Gemma4ChatTemplateParser(#1551), cross-layer KV cache sharing (#1896), chunked prefill with prefix KV cache reuse (#1906), Pallas Splash attention (#1890), and distributed RL support for Gemma 4 E2B (#2347, #3364). - Qwen 3.5 & Qwen 3: Added native Qwen 3.5 hybrid models (
0.8B,2B) with Flash Attention, sequence packing, and chunkwise Gated DeltaNet TPU optimization (#3088, #3096), plus Qwen 3.5 35B/397B-A17B configs, vLLM mappings, and distributed RL recipes (#1799, #1834, #3182, #3379) and Qwen 3 0.6B (#2064). - Raiden Weight Sync & Fault Tolerance: End-to-end Raiden FFI/Pathways weight sync across Tunix and MaxText trainers with multi-host discovery and MoE 128-lane TPU GMM interleaving (#2059, #2096, #2146–#2151, #2201, #2970, #3032), plus rollout worker fault eviction, automatic re-dispatch, late-rejoin bring-up, and capacity-floor guardrails enabled by default (#2722, #3050, #3191, #3213).
- JAX GPU & v2 Continuous-Batching Sampler (
vanilla_v2): Added single- and multi-host JAX GPU support with Tokamax FlashAttention/RaggedDot and vendored vLLM Triton paged attention (#3010, #3101, #3102, #3265), plus the experimental v2 continuous-batchingLLMEnginewith multi-rank DP, prefix caching, async scheduling, and 1.75× faster unconstrained decoding (#2917–#2929, #3051, #3057, #3272). - Sequence Packing & High-Throughput Data Plane: Unified sequence packing with a C++
nanobind(nogil,abi3) extension (#1673, #2135, #2720, #2723), MoE router replay with compactuint8/uint10bit-packing and Orchestrator Relay (#2046, #2818, #3271, #3300), and out-of-band bulk TCP + parallel LZ4 tensor transport (#2774, #2776, #2817). - Trajectory Store & Observability: Added ATIF-backed In-Memory, File, and SQL (SQLite/PostgreSQL partitioned) trajectory & metrics stores (#1857, #2206, #2753, #2984, #2985, #3246),
trajectory-exploreCLI (#2282, #2933, #2953), OpenTelemetry worker metrics (#2174, #3132, #3240), and Google Cloud ML Diagnostics backend (#2165, #3085).
API Changes
- Renamed
RLCluster→RLEngineacross the codebase (#1810) and graduatedtunix.experimentaldistributed RL modules totunix.dist_rl(#2999). - Removed
sglang_jaxrollout/sampler support;vanilla(vanilla/vanilla_v2) andvllmare now the supported rollout backends (#3204). - Removed DeepSWE OpenHands scaffold (
r2egymandsweagentremain supported, #2761) and removed MLPerf RCP (mllog) integration (#2762). - Standardized multi-turn agentic RL on Token-In-Token-Out (TITO), removing the legacy re-tokenization path (
exact_token_continuity, #3254). - Config & Interface Updates:
GRPOConfig.num_generationsis now the single spelling for GRPO group size (#2203, #2237);max_seq_token_per_tpumigrated tomax_seq_tokens_per_device(#3284, #3322).- Renamed
AlgorithmConfig.reward_manager→reward_mode("reward_fn"/"env", #3046) and replacedforce_on_policy_ratio/log_sampler_trainer_agreementwith unified in-loss agreement fusion &sampler_is/sampler_rs(#2578, #2616). AbstractTrainernow uses explicit args instead of**kwargs(#3019) and callback-basedwith_fwd_only_fn/fwd_only(payload)(#2273, #3225, #3243).RolloutWorker.sample_prompts()removed;generate()requiresRolloutRequest(#2476), andRolloutWorker/RolloutManagerrequire explicitconfig,tokenizer,sampler, andchat_parser(#3074, #3347).InprocessVllmSampler/VllmSamplernow build engine configs internally fromRolloutConfig(#3170);cluster_config.k8s.trainer_extra_envreplaced bycluster_config.role_env.<role>(#3318).
- Dependencies: Updated
flax>=0.12.5(#1581), added JAX0.11.0support (#1963), addedgymnasium(#1613) andsqlalchemy(#2204), and added C++nanobindwheel build (requires-python >= 3.12, #2723).
What's Changed
- Algorithms & Training: Enabled PPO across all model types (#1683, #1752, #1775), added model-agnostic JAX diffusion contracts (#1832), ORPO TRL parity (#1574), DAPO
beta=0.0(#2570), chunked logps (#1595), mixed-precisionfp32master /bf16compute training (#2844, #2972), and Orbax V1 Checkpointer migration for SFT (#1571, #1679, #1690). - Rollouts & Agentic RL: Added partial rollouts (in-flight weight updates, #2750), prompt-arrival group ordering &
max_version_spanstaleness enforcement (#2725, #3104, #3165), DeepSWE Agent Sandbox & SandboxFleet dynamic pre-warming (#1791, #2125, #2189, #3161), process-pool reward evaluation (#1904), and Qwen 3.5<tool_call>parsing & timeout fixes (#2820, #2821, #2934). - Performance, Serialization & CI: Reduced optimizer/gradient HBM footprint (#1758, #1783), added Tokamax Splash Attention (#1943, #2962), GIL-free C++ trajectory serialization & compact JSON (#2731, #2739),
safetensorsthread-safety &bfloat16fixes (#1627, #2016, #3139), notebook fixes (#1738–#1741), and faster sharded CPU/TPU CI (#2272, #3155, #3166, #3173).
New Contributors
- @yatlas20098 made their first contribution in #1618
- @AntonyMei made their first contribution in #1636
- @fallintoplace made their first contribution in #1629
- @AMR5210 made their first contribution in #1627
- @deep-diver made their first contribution in #1570
- @kitrakrev made their first contribution in #1552
- @ethannnnnn made their first contribution in #1833
- @gutianyu-google made their first contribution in #2088
Full Changelog: v0.1.7...v0.1.8