Repository navigation
Releases
Releases · vllm-project/vllm-torchtpu
Release v0.33.0
vllm-torchtpu v0.33.0
Basic Info
- Cut: 2026-11-12 (branch point
008ca4a41onmain), 2026-11-13 (cherry-picked wheel build fix for the release pipeline, #3215). - Tag commit:
46ae4b7a6(v0.33.0,releases/v0.33.0). - vLLM:
v0.33.0(wasv0.32.0at v0.32.0); vllm-omni0.32.0rc1(unchanged). - Compared to:
releases/v0.32.0-- 134 PRs (excluding 8 automated support-matrix and verified-commit-hash bot commits, and 2 PRs that already shipped in v0.32.0 as cherry-picks; see Note).
Highlights
- vLLM 0.33.0 (#2693) -- the plugin moves to vLLM 0.33.0 and adopts its API changes: KV cache planning moves to
vllm.v1.core.kv_cache_planning,rayalways usesRayDistributedExecutorV2, and draft KV specs are taggedis_draft. - Multi-LoRA serving (#2360, #2359, #2376, #2378, #2501) --
--enable-loranow works for dense text models on a single TPU chip. Requests in one batch can each use a different adapter or none, and adapters load, swap and evict without recompiling. Each swap writes the adapter slot with one TPU program instead of about 790 small writes. - 128-token KV pages for dense long-context models (#2425) -- dense
FLASH_ATTNmodels withmax_model_len > 8192now get 128-token pages instead of 16-token pages: +47.3% total tok/s on Qwen3-32B TP=8 (8k/1k, c=64), and 3.6x total tok/s on a Raiden KV-offload prefix-reuse benchmark on v7x-8. - Faster GDN prefix caching (#2169, #2937, #2783) -- internal prefill checkpoints let a prefill chunk run to the end of the prompt instead of stopping at the last cacheable block. This cuts mean TTFT by 11% to 27% on Qwen3.5-397B-A17B-FP8 PP4xPCP8 / TP4xPCP8 traces and is on by default. GDN state is now correct at block boundaries under MTP, and compact-mamba engines no longer charge GDN groups for scheduler blocks.
- Static EPLB (#1408) --
TPU_EXPERT_PLACEMENTloads a per-layer expert placement table and applies it as a load-time permutation, so every MoE kernel serves it unchanged. On DeepSeek-V4-Pro AgentX (v7x-16, DP=16, EP=16, c=160): p90 TTFT -12.0%, total throughput +2.6%, accuracy unchanged. - Faster cold start (#2239, #2647, #1715, #1900, #2452, #2600, #2601) --
TPU_PRECOMPILE_TRACE_FIRSTtraces every bucket group first and then compiles all buckets at once, cutting GLM-5.2 AgentX compile time from 99 min to 50 min. Peer replay now works across DP/TP/PCP/DCP subgroups (-17.5% cold-start compile on Qwen3.5-397B-A17B-FP8 DP4xTP2), and sampler and rejection-sampler sweeps rotate across ranks (rejection-sampler sweep 97 s to 55 s). - Opt-in buffered pipeline hand-off (#2527) -- with
TPU_PP_HANDOFF=ring, each stage's output goes straight to the next stage instead of moving in lockstep: +13.9% to +20.4% input tok/s at 1 to 4 streams on Qwen3.5-35B-A3B-FP8 PP4xPCP8 (four v7x hosts), and up to +44.8% on multi-turn sessions. - TPU8i bring-up (#2514, #2515, #2516, #2689) -- GDN PCP prefill admits TPU8i. The engine starts with a
libtpu.soloaded throughTPU_LIBRARY_PATHwithout the pip package. SparseCore kernels run only on chips they are validated on, and FP8 models load on torch_tpu builds newer than 0.2.0.
New features
Capabilities
- Multi-LoRA (
--enable-lora): runner, worker and KV cache budget wiring with an e2e test (#2360); TPU LoRA managers that write an adapter slot with one program and reject non-finite adapters (#2359); single-program slot write op (#2376); on-chip manager tests (#2378); docstring giving the per-layer cost of the shrink/expand ops in terms ofmax_lorasandmax_lora_rank(#2501). Limits: dense text models on one chip, no DP/TP/PP/CP or speculative decoding (startup check from v0.32.0), and the LoRA ops slow down requests without an adapter too. - DSpark adaptive draft verification, opt-in with
"enable_adaptive_verification": truein the DSpark speculative config: confidence head loaded from DeepSeek-V4 DSpark checkpoints (#2592), per-draft confidences from the draft step (#2593), draft budget, trim and step-cost model (#2594), runner wiring with profiling, warmup and rollback (#2531). Outputs are unchanged. On DeepSeek-V4 Flash (DP8) and Pro (DP16), throughput was within about 1% of the baseline. - Static EPLB:
TPU_EXPERT_PLACEMENT(local path orgs://JSON table) permutes experts and router at load for every MoE kernel, target and drafter (#1408) - DP vocab parallelism:
TPU_DP_VOCAB_PARALLEL=1(off by default) shards the embedding and lm_head over every DP x TP rank instead of each engine's TP ranks: 3.79 GiB to 0.24 GiB per device for Qwen3.8-2.4T at DP16 x TP2 on TPU8i. Needs DP + EP, and falls back to the TP split for PP, CP, speculative decoding, LoRA, pooling and multimodal models (#2863) - BF16 MoE experts quantized to FP8 on load:
QUANTIZE_ON_LOAD_PREFIXESnow coversRoutedExperts, e.g.QUANTIZE_ON_LOAD_PREFIXES=expertsquantizes only the routed experts of a BF16 checkpoint (#2967) - Pluggable TPU attention backends: plugins can register a backend name that
--attention-backend <name>selects; groundwork for the FlyWheel backend (#2271) - TPU8i: GDN PCP ragged exchange admits TPU8i (#2514); a
libtpu.soloaded throughTPU_LIBRARY_PATHwithout thelibtpupip package is tolerated and used for the compile cache key and version check (#2515); SparseCore gather/scatter/reduce kernels gated on an allowlist of validated chips, so other chips take the TensorCore fallback (#2516);fresh_buffercopies float8 through bfloat16, so FP8 models load on torch_tpu builds after 0.2.0 (#2689) - RL integration:
TPUWorker.reset_encoder_cacheimplemented, soEngineCore.reset_encoder_cache()after a weight update no longer fails (#2732, #2771); MoE top-k routing weights kept in FP32 withTPU_MOE_TOPK_WEIGHTS_FLOAT32(off by default) to lower trainer-vs-sampler logprob mismatch (#2673) - P/D and offload: Raiden decode replicas with TP4 (#2339); asynchronous Raiden decode load failures go through vLLM's native
failed_recvingpath, with opt-in prefill send-completion signaling viaadditional_config.raiden_pd_send_completion(#2302);TPU_RAIDEN_TRANSFER_STREAMSsizes the TCP streams per push independently of the transfer rank count (#1911) - Startup validation and diagnostics: prefix caching on attention-free models is rejected at startup (#2597); set environment variables that nothing reads are reported at startup with a suggested name, and valid
VLLM_TPU_*knobs no longer fail--fail-on-environ-validation(#2323) - Experimental AOT export: MTP draft export, verify windows, debug taps, KV paging at the attention kernel's block size, Gemma 4 support, and the model's own
forward/compute_logitscaptured (#2709); KV cache groups for mixed sliding-window and full-attention models, a generic drafter name, the model id recorded asmodel_name, and the serving checkpoint written as Zarr v2 arrays in OCDBT (#2700)
Groundwork (not yet user-facing in this release)
- Megakernel framework: SPMD helpers,
PeerTopologyandRemoteCopyBatch(#2363);ops/anddma/packages with lazy init (#2712); runtime contract and step statistics (#2713); plugin-facing runner views and hooks (#2715);VLLM_TPU_MEGAKERNELconfig parsing and plugin fingerprint (#2716); bucket policy and step eligibility (#2908); rank-uniform decline agreement and startup guards (#2806);StepRoutingper-step inspection and routing (#2948); pool layout updates with FP32 conv-state support (#2788); core spec, planner and profile registries with lazy builtins (#2797);megakernel/moved tomegakernel/core/andmegakernel_runtime/tomegakernel/adapter/(#3019) - Model directory scaffolds under the Experimental tier: Kimi-K3 / Kimi-K3.1-Pro, home of the megakernel migration (#2837); MiniMax-M3.1-Flash (#2767)
Performance and kernel improvements
Attention / MLA
- 128-token KV pages for dense
FLASH_ATTNmodels withmax_model_len > 8192, 8x fewer Raiden offload transfers (+47.3% tok/s on Qwen3-32B TP=8 8k/1k c=64; +56.2% on Llama-3.1-8B TP=8 at 131K context) (#2425) - Sparse MLA packs the FP8 RoPE cache to 64 B/token by default, 576 instead of 640 B/token: 10.4% more tokens in the same HBM on GLM-5.2, +2.7% to +4.0% TPS/chip at c=48 and c=64 (#2206)
- PCP8 streaming RPA ring uses 3-slot KV buffering: kernel latency -8.7% to -15.6% on v7x, +1.8% to +5.5% tok/s end to end on PP4 x PCP8 (#2263)
TPU_MLA_SPARSE_SPLIT_Qis honored in non-DCP sparse MLA, where it was silently ignored (#2524)
MoE
- Fused MoE v2 decode, parts 1 and 2 of 5: the next expert's token rows are issued under the current tile's compute (#3074), and the short height rung is placed at the expected expert load (T64 292.6 to 272.9 us, bit-exact) (#3075). See Note.
- MoE routing prep uses one counting sort for EP one-hot and non-EP TP paths up to 2,048 rows: 25% to 63% less routing-prep time per layer on v7x and v6e, bit-identical (#2604)
- Qwen3.5 MoE attention/MoE TP collectives overlapped with reduce-scatter and all-gather on large prefill buckets, opt-in with
TPU_ATTN_MOE_OVERLAP=True(#1492)
GDN / hybrid
- Internal prefill checkpoints for GDN prefix caching in Qwen3.5 and Qwen3.8, on by default;
TPU_GDN_PREFILL_CHECKPOINT=0turns it off (#2169) - GDN groups no longer take scheduler blocks on the compact-mamba layout (Qwen3.8-2.4T admits 93 instead of 89 requests per engine from a 6,720-block pool); on by default,
TPU_SKIP_MAMBA_SCHEDULER_BLOCKS=0opts out (#2783) - Opt-in (k, n) layout for
in_proj_qkvz,TPU_GDN_QKVZ_KERNEL_LAYOUT=1: decode 119.1 to 110.9 ms per step on a 69-GDN-layer per-block FP8 model at DP16 x TP2 (#2719) - GDN single-token decode fuses the Q/K projection: pea...
Release v0.32.0
Basic Info
- Cut: 2026-10-29 (branch point
c7f724b2donmain), 2026-10-30 (two cherry-picked release-pipeline fixes, #2522 and #2634). - Tag commit:
1e5cd9e95(v0.32.0,releases/v0.32.0). - vLLM:
v0.32.0(wasv0.29.0at v0.29.0); vllm-omni0.32.0rc1(was0.29.0rc1). - Compared to:
releases/v0.29.0-- 230 PRs (excluding 12 automated support-matrix and verified-commit-hash bot commits, and 8 PRs that already shipped in v0.29.0 as cherry-picks; see Note).
Highlights
- vLLM 0.32.0 (#1850, #2183) -- the plugin moves from vLLM 0.29.0 through 0.31.0 to 0.32.0, with vllm-omni 0.32.0rc1 and
transformers>=5.18.0. Hybrid models keep kernel block ids in split KV cache groups, which vLLM 0.32 no longer provides. - Qwen4Exp (Qwen3.8-Flash-Next) on TPU (#1903, #1926, #1974) -- a native TPU text model with hyper-connection residuals that loads the released FP8 checkpoint, including the n-gram PLE layer.
- Kimi-K3 DSpark speculative decoding and P/D (#1711, #1712) -- DSpark drafting over pooled KDA state, then prefill/decode disaggregation for it (GSM8K 97.65%, acceptance length 4.41 on v7x-32).
- Experimental AOT export and Rust serving runtime (#2260, #2277, #2380, #2445, #2518, #2449) --
python -m vllm_torchtpu.experimental.exportwrites a vLLM model out as TorchTPU StableHLO serving artifacts;python -m vllm_torchtpu.experimental.rust serveserves them with the Rust runtime behind thevllm servecommand line. - Opt-in fused MoE decode kernels (#1678, #1763, #1930) -- one Pallas call per rank for the routed experts, shared expert and all-reduce at low concurrency; the GLM-5.2 TP variant cuts P90 TPOT by 14.7% on an AgentX replay.
- DeepSeek-V4 performance (#2377, #2161, #1854, #1729, #1982, #2290) -- kernel tuning for HCA/CSA attention, the compressor and the indexer (about 4% on AgentX c=160 with DSv4 Pro), SparseCore top-k overlapped with TensorCore work, and attention weights sharded across chiplets.
- SRPT prefill scheduler (#1924, #2252) -- an opt-in scheduler class that orders chunked prefill by shortest remaining work, so long prefills no longer block short or prefix-cached requests, with an opt-in attention work cap for data-parallel decode.
- Diffusion (vLLM-Omni) speedups (#1432, #1452, #1448, #1450) -- a Pallas flash-attention backend is now the default for diffusion transformers, plus a fused q/k RMSNorm + RoPE kernel and SparseCore all-to-alls for Wan.
- Published on PyPI and Docker Hub (#1618, #2204, #2522, #1721) --
vllm-torchtpu-coreandvllm-torchtpuwheels with PyPI trusted publishing and PEP 740 attestations; building no longer needs private registry access.
New features
Model support
- Qwen4Exp (Qwen3.8-Flash-Next): native TPU text model with hyper-connection residuals, GDN linear attention and Qwen3Next sparse MoE (#1903); loads the released FP8 checkpoint as a text model (#1926); n-gram PLE layer and its weights (#1974)
- Qwen2-family GQA models at TP=8 when the query-head count does not divide by TP, by zero-padding Q heads per KV group (Fin-R1-7B on v7x-8) (#1739)
- Kimi-K3 + DSpark: speculative decoding with pooled KDA state (#1711) and prefill/decode disaggregation (#1712)
- DeepSeek-V4: static-q speculative verify pass for full-length verify windows (#2129); attention
wq_b/wo_a/wo_bsharded across chiplets (#1982); the compressor writes the indexer K cache inSEQ_ALONG_LANE(#2225); the HCA compressor state is excluded from prefix caching and KV offload (#2389) - Qwen3.5 MTP: prefill-only MTP on a PPxPCP producer for disaggregated speculative decoding (#1889)
- GLM P/D: TP/DCP prefill to TP/DCP decode KV transfer, each prefill DCP rank sending its pages to the matching decode rank (#2211)
- NVFP4 checkpoints fall back to
ModelOptLinearMethod(#1257)
Diffusion (vLLM-Omni)
- Pallas flash-attention backend for diffusion transformers, now the default (
TPU_FLASH_ATTN); SDPA stays selectable and is the fallback for causal, masked or non-bf16 calls (#1432) - TPU device profiler for diffusion workers (#1428)
- Wan: fused q/k RMSNorm + RoPE Pallas kernel (#1452), pair-swap matmul RoPE (#1448) with 2-D token-major cos/sin tables for vllm-omni 0.32 (#2226), GELU fenced from the down-projection fusion (#1449), timestep materialized before the transformer (#1451)
- Dense all-to-alls offloaded to SparseCore in diffusion workers (#1450)
- LTX-2: split RoPE applied on the q/k layout with reused tables (#2298)
- vllm-omni pinned to 0.31.0rc1 with obsolete TPU patches dropped (#1945), then 0.32.0rc1 (#2183); libgl added to the omni image for multi-rank Ulysses (#1939)
Capabilities
- Experimental AOT export to TorchTPU StableHLO serving artifacts (#2260): tensor-parallel programs, Orbax weight checkpoints, hybrid (GDN) models and FP8 KV cache (#2277); one
serving_configinmetadata.json(#2380); serve CLI arguments, speculative config, sort-free samplers and more backbones (#2445); the XLA flags libtpu runs with are recorded, and weight loading and saving are traced (#2518) - Experimental Rust runtime launcher:
python -m vllm_torchtpu.experimental.rust serve <model> [vllm serve args]exports once and execsrvllm serve(#2449) - SRPT prefill scheduler (
--scheduler-cls vllm_torchtpu.core.experimental.srpt_prefill_scheduler.SrptPrefillScheduler) (#1924), with an opt-in attention work cap,VLLM_TPU_SRPT_ATTN_WORK_CAP(#2252) - CUDA-matching RMSNorm and RoPE numerics: a
tpuvLLM IR op provider forrms_norm/fused_add_rms_normand out-of-tree RoPE CustomOps that compute in FP32 as the CUDA kernels do, to lower trainer-vs-rollout KL in RL (#2312) - RL sampling precision: logprob normalizer computed in FP32 without a full-vocab
log_softmax(#2024), logprob quantization fixed (#1745), Gumbel-Max noise drawn in FP32 (#1747) - Offload and P/D: Raiden offloading connector admitted under pipeline parallelism (#2156); GLM-5.2 SparseCore MLA caches admitted in Raiden reshard (#1978); DCP fine prefix hits kept with a CPU offloading connector (#2040); PCP/TP/DP producer KV blocks registered ahead in the prompt-completing step (#1813);
least_loadeddecode routing in the disagg toy proxy (#2195) - Speculative decoding:
RejectionSampleraccepts an optional draft distribution for standard speculative sampling (groundwork; no caller passes it yet) (#2439); DSpark warns instead of erroring on config mismatches (#2187); better DSpark first-pass warmup (#1979) - Startup validation: the platform rejects a per-step token budget that cannot fit the largest
compile_sizesentry, and non-integercompile_sizes, instead of crashing on the first large step (#1737) TPU_SPARSE_MLA_CHUNK_TOKENSsets the sparse MLA chunk loop size (#2035)
Groundwork (not yet user-facing in this release)
- Multi-LoRA: startup check for unsupported LoRA settings, LoRA in bf16 (#2372, #2418); shrink/expand ops for mixed-adapter batches (#2358); TPU Punica wrapper (#2375); recompile-free swap test (#2374); dtype-mismatch message under
torch.compile(#2462). See Note:--enable-lorais not wired up in v0.32.0. - TurboQuant KV cache [1/3]: 128-lane KV page layout, quantization and VREG dequantization primitives; no existing code path uses them yet (#2228)
- Megakernel framework (
megakernel/): error hierarchy and numeric primitives (#2209), VMEM pool aliasing (#2227), execution config and default tiles (#2276), raw-pool layout and paged cache config (#2286), declarative model spec (#2292), raw-pool packing utilities (#2305), module options and declarations (#2318), pool backend factory (#2324), Pallas pool recurrent-state gather (#2361) and KV/checkpoint scatter (#2382) kernels, Block renamed to Module (#2390), execution profile registry (#2409), declaration-driven planner (#2448),PoolLayoutderived from module cache declarations (#2476); fixes for redundant BF16 truncations (#2241) and unknown tunables / unvalidated chips (#2284)
Performance and kernel improvements
Attention / MLA
- Batched RPA: seq-on-lane 128-byte aligned and a triple-buffered pipeline with in-step KV writeback (+0.6% to +1.3% throughput on Qwen3.5-397B-A17B-FP8, v7x-8) (#1420)
- v2 ragged paged attention returns the LSE for MLA (#1639)
- MLA DCP q-replicate uses a rank-independent dead q shard, so precompile peer replay works (#2190)
MoE
- Fused MoE decode kernel for low concurrency, opt-in with
FUSED_MOE_DECODE_KERNEL=1(#1678) - GLM-5.2 fused TP decode MoE kernel with in-kernel routing, shared expert and all-reduce, opt-in with
USE_MOE_TP_DECODE_KERNEL=1; 36% to 60% less time per decode MoE layer than EP16 on v7x TP16 (#1763) - Fused EP MoE: device-id ordered activation all-gather under DCP and EP, opt-in with
MOE_FUSED_EP_SORTED_GATHER(#1930);uniform_randomrouting simulation honoured instead of disengaging (#1505); fp4 scale tables staged as fresh base tensors (#1857)
GDN / hybrid
- GDN mixed and decode tile sizes unified and tuned to the memory budget (#1563)
- Unified-pool GDN refactor: dense and pooled state share one granule-addressed path; pooled conv state defaults to FP32 (#1620)
DeepSeek-V4
- HCA/CSA attention, compressor and indexer kernel tuning, bit-exact, about 4% on AgentX c=160 (#2377)
- Indexer SparseCore top-k overlapped with TensorCore work (#2161)
- Kernel block sizes tuned for long-context / AgentX workloads (#1854)
- Transposed compressor weights materialized once instead of cloned every step (#1729)
- StreamIndex: pass schedules shared across indexer layers and built in O(segments + steps) (#2290); small-bucket MIXED-pass blo...