v0.24.0

• Pypi
• Docker

Highlights

• DeepSeek-V4 bringup — initial main attention kernels, DeepseekV4Attention wiring, KV cache creation, KV compressor + quick optimizations, indexer, and MHC HCHeadOp support land the model end to end under experimental/deepseek_v4 (#2903, #2980, #2981, #2858, #2993, #2905, #2950, #2986).
• dFlash speculative decoding — new DFlash model + proposer, relocated into vLLM with added unit tests and split CI coverage (#1868, #2969, - #2904, #2939).
• SparseCore MoE kernels — SC dense gather-reduce kernel, ragged_gather_v2 / ragged_gather_reduce_v2, GMM cleanup, and cost-model tiling/- partitioning refinements for MoE layers (#2979, #2836, #2985, #2928, #2887).
• gdn/v3 fused Conv1D/GDN kernel — new fused Conv1D + gated delta-rule (GDN) kernel for gated-attention models, with reference implementations - and kernel tests (#3016).
• Raiden KVCacheManager migration — TPUConnector and the KV cache manager move to Raiden, with import-path completion and a load-order XLA-symbol - crash fix (#2868, #2997, #3025, #2886).
• Multihost Ray Executor V2 — multi-host TPU now runs on vLLM's Ray Executor V2 via the native VLLM_USE_RAY_V2_EXECUTOR_BACKEND flag, with - serve-log dumping and Buildkite step short-circuiting (#2893, #2971, #2976, #2902).
• Gemma 4 improvements — Jax QKV fusion, num_heads o_proj sharding, and mm-encoder-tp-mode recipes (#2908, #2938, #2892).
• JAX 0.10.2 upgrade — toolchain bumped to JAX 0.10.2, plus a SparseCore lowering fix for the 0.10.1 regression (#2949, #2887).
• Fused QKV projections — fused QKV reduces redundant HBM-to-VMEM data loading on the attention path (#2865).

New features

Model support

• DeepSeek-V4 main attention kernels and DeepseekV4Attention, with KV cache creation, KV compressor, indexer, core attention, MHC HCHeadOp - support; code lives under experimental/deepseek_v4 (#2903, #2980, #2981, #2858, #2993, #2905, #2950, #2986).
• dFlash spec-decoding model and proposer, relocated alongside the torchax dflash model / spec_decode wrapper in vLLM (#1868, #2969, #2904).
• Gemma 4 gains Jax QKV fusion, num_heads o_proj sharding, and mm-encoder-tp-mode recipes (#2908, #2938, #2892).
• Llama 4 Maverick rope_scaling factor-naming and jax-convert OOM fixes (#3003).

Capabilities

• gdn/v3 fused Conv1D/GDN kernel for gated-attention models, with reference impls and kernel tests (#3016).
• KV offloading for attention data parallelism (#2840).
• KV connector queue lengths exposed as metrics (#2886).
• MMEncoderJITManager re-implemented for the torchax multimodal path (#2802), with the Qwen3.5 vision-tower patch removed (#2878).

Performance & kernel improvements

Attention / MLA

• Fused QKV projections cut redundant HBM-to-VMEM data loading (#2865).
• Removed with_layout_constraint from batched-RPA prepare_inputs (#2918).
• Dropped unused attention metadata fields (#2943).

GDN

• gdn/v3 fused Conv1D/GDN kernel (fused Conv1D + ragged gated-delta-rule) with reference implementations and kernel tests (#3016).

MoE / DP

• SC dense gather-reduce kernel for MoE layers (#2979) plus refined cost-model tiling/partitioning (#2985).
• SparseCore ragged_gather_v2 + ragged_gather_reduce_v2 MoE kernels (#2836).
• MoE unpermute pipelining with collectives via VLLM_MOE_CHUNK_SIZE (#2869).
• GMM kernel cleanup (#2928).
• Single-mesh SparseCore kernels lowered through core_map to fix the Jax 0.10.1 regression (#2887).

Quantization

• Per-block clipping in MoE weight requantization (#2961) and MOE_REQUANTIZE_CLIP_PERCENTILE for FP4 requantization (#2894).
• GMMv2 binds weight precision to LHS dtype in the dequantize-before-matmul path (#2891).

Caching

• Removed jax.clear_caches() from assign_and_shard_param to avoid recompilation and speed up weight loading (#2974).

Misc

• Kernel-tuner specific flags (#2834) and more variables in the TuningKey (#3006).
• xla autotune OFAT sweep over LIBTPU_INIT_ARGS on Buildkite (#2807).

Bug fixes

Correctness

• Qwen3-30B-A3B accuracy crash on v6e — SC dense_gather_reduce now falls back when the output block has zero rows (#3036, shipped via #3044).
• Multimodal "TPU already in use" failure — multimodal test releases the TPU between parametrizations (#3037, shipped via #3044).
• Raiden XLA symbol crash — load tpu raiden before jax (#3025).
• Routed-experts slot mapping for prompt tokens corrected (#2965).
• FP8 Compressed Tensors load AttributeError on TPU fixed (#2879).
• rope_scaling values now read correctly from config (#2838).
• Llama 4 Maverick rope_scaling naming + jax-convert OOM fixed (#3003).
• fastAPI version restricted to fix a compatibility break (#2890).

Infra, CI, docs

• vLLM integration error fixes and LKG version bumps (#2962, #2951, #2933).
• Multihost: enable Ray Executor V2 via native VLLM_USE_RAY_V2_EXECUTOR_BACKEND flag (#2971); dump vllm_serve.log in run_multihost.sh cleanup - (#2976); short-circuit the Buildkite step so it doesn't block waiting for TPUs (#2902).
• Raiden KV cache manager import path completed (#2997).
• DP: example external round-robin LB (#2934) and PHASED_PROFILING_DIR collision fix in run_dp_external_lb.sh (#2945).
• Build docker image for daily benchmarking (#2932); build the image once instead of per-pipeline (#2883); skip empty metric values in - mlcompass_export (#3014).
• Benchmarking script: example NVFP4/TPU-FP4/FP8 perf benchmarking for Qwen3.5-397B (#2912).
• CI: split spec-decoding e2e into per-push BVT subset + nightly full matrix (#2939); fix Kimi-K2.6 nightly OOM + mmlu --trust-remote-code - (#2923); fix mmlu continue-decode num-tokens (#2931); fix flaky Qwen3-MoE weight-loading memory test on tpu6e (#2996); enable - processed_logprobs fixture mode (#2916); resolve dp scheduler test signature/unused-var (#2911); non-ready pipeline steps run on CPU not TPU - (#2914); nightly_verify skips commit-back on release-branch nightlies (#2941).
• Test teardown: explicit server shutdown instead of dropping the reference and waiting (#2882), applied across remaining test files (#2915).
• torchax bumped to 0.0.13 (#2917).
• Docs: v0.23.0 release docs (#2966) and support-matrix table updates (#2929).