Skip to content

Releases

Release v0.33.0

Choose a tag to compare

@charlesli640 charlesli640 released this 17 Nov 17:34

vllm-torchtpu v0.33.0

Basic Info

  • Cut: 2026-11-12 (branch point 008ca4a41 on main), 2026-11-13 (cherry-picked wheel build fix for the release pipeline, #3215).
  • Tag commit: 46ae4b7a6 (v0.33.0, releases/v0.33.0).
  • vLLM: v0.33.0 (was v0.32.0 at v0.32.0); vllm-omni 0.32.0rc1 (unchanged).
  • Compared to: releases/v0.32.0 -- 134 PRs (excluding 8 automated support-matrix and verified-commit-hash bot commits, and 2 PRs that already shipped in v0.32.0 as cherry-picks; see Note).

Highlights

  • vLLM 0.33.0 (#2693) -- the plugin moves to vLLM 0.33.0 and adopts its API changes: KV cache planning moves to vllm.v1.core.kv_cache_planning, ray always uses RayDistributedExecutorV2, and draft KV specs are tagged is_draft.
  • Multi-LoRA serving (#2360, #2359, #2376, #2378, #2501) -- --enable-lora now works for dense text models on a single TPU chip. Requests in one batch can each use a different adapter or none, and adapters load, swap and evict without recompiling. Each swap writes the adapter slot with one TPU program instead of about 790 small writes.
  • 128-token KV pages for dense long-context models (#2425) -- dense FLASH_ATTN models with max_model_len > 8192 now get 128-token pages instead of 16-token pages: +47.3% total tok/s on Qwen3-32B TP=8 (8k/1k, c=64), and 3.6x total tok/s on a Raiden KV-offload prefix-reuse benchmark on v7x-8.
  • Faster GDN prefix caching (#2169, #2937, #2783) -- internal prefill checkpoints let a prefill chunk run to the end of the prompt instead of stopping at the last cacheable block. This cuts mean TTFT by 11% to 27% on Qwen3.5-397B-A17B-FP8 PP4xPCP8 / TP4xPCP8 traces and is on by default. GDN state is now correct at block boundaries under MTP, and compact-mamba engines no longer charge GDN groups for scheduler blocks.
  • Static EPLB (#1408) -- TPU_EXPERT_PLACEMENT loads a per-layer expert placement table and applies it as a load-time permutation, so every MoE kernel serves it unchanged. On DeepSeek-V4-Pro AgentX (v7x-16, DP=16, EP=16, c=160): p90 TTFT -12.0%, total throughput +2.6%, accuracy unchanged.
  • Faster cold start (#2239, #2647, #1715, #1900, #2452, #2600, #2601) -- TPU_PRECOMPILE_TRACE_FIRST traces every bucket group first and then compiles all buckets at once, cutting GLM-5.2 AgentX compile time from 99 min to 50 min. Peer replay now works across DP/TP/PCP/DCP subgroups (-17.5% cold-start compile on Qwen3.5-397B-A17B-FP8 DP4xTP2), and sampler and rejection-sampler sweeps rotate across ranks (rejection-sampler sweep 97 s to 55 s).
  • Opt-in buffered pipeline hand-off (#2527) -- with TPU_PP_HANDOFF=ring, each stage's output goes straight to the next stage instead of moving in lockstep: +13.9% to +20.4% input tok/s at 1 to 4 streams on Qwen3.5-35B-A3B-FP8 PP4xPCP8 (four v7x hosts), and up to +44.8% on multi-turn sessions.
  • TPU8i bring-up (#2514, #2515, #2516, #2689) -- GDN PCP prefill admits TPU8i. The engine starts with a libtpu.so loaded through TPU_LIBRARY_PATH without the pip package. SparseCore kernels run only on chips they are validated on, and FP8 models load on torch_tpu builds newer than 0.2.0.

New features

Capabilities

  • Multi-LoRA (--enable-lora): runner, worker and KV cache budget wiring with an e2e test (#2360); TPU LoRA managers that write an adapter slot with one program and reject non-finite adapters (#2359); single-program slot write op (#2376); on-chip manager tests (#2378); docstring giving the per-layer cost of the shrink/expand ops in terms of max_loras and max_lora_rank (#2501). Limits: dense text models on one chip, no DP/TP/PP/CP or speculative decoding (startup check from v0.32.0), and the LoRA ops slow down requests without an adapter too.
  • DSpark adaptive draft verification, opt-in with "enable_adaptive_verification": true in the DSpark speculative config: confidence head loaded from DeepSeek-V4 DSpark checkpoints (#2592), per-draft confidences from the draft step (#2593), draft budget, trim and step-cost model (#2594), runner wiring with profiling, warmup and rollback (#2531). Outputs are unchanged. On DeepSeek-V4 Flash (DP8) and Pro (DP16), throughput was within about 1% of the baseline.
  • Static EPLB: TPU_EXPERT_PLACEMENT (local path or gs:// JSON table) permutes experts and router at load for every MoE kernel, target and drafter (#1408)
  • DP vocab parallelism: TPU_DP_VOCAB_PARALLEL=1 (off by default) shards the embedding and lm_head over every DP x TP rank instead of each engine's TP ranks: 3.79 GiB to 0.24 GiB per device for Qwen3.8-2.4T at DP16 x TP2 on TPU8i. Needs DP + EP, and falls back to the TP split for PP, CP, speculative decoding, LoRA, pooling and multimodal models (#2863)
  • BF16 MoE experts quantized to FP8 on load: QUANTIZE_ON_LOAD_PREFIXES now covers RoutedExperts, e.g. QUANTIZE_ON_LOAD_PREFIXES=experts quantizes only the routed experts of a BF16 checkpoint (#2967)
  • Pluggable TPU attention backends: plugins can register a backend name that --attention-backend <name> selects; groundwork for the FlyWheel backend (#2271)
  • TPU8i: GDN PCP ragged exchange admits TPU8i (#2514); a libtpu.so loaded through TPU_LIBRARY_PATH without the libtpu pip package is tolerated and used for the compile cache key and version check (#2515); SparseCore gather/scatter/reduce kernels gated on an allowlist of validated chips, so other chips take the TensorCore fallback (#2516); fresh_buffer copies float8 through bfloat16, so FP8 models load on torch_tpu builds after 0.2.0 (#2689)
  • RL integration: TPUWorker.reset_encoder_cache implemented, so EngineCore.reset_encoder_cache() after a weight update no longer fails (#2732, #2771); MoE top-k routing weights kept in FP32 with TPU_MOE_TOPK_WEIGHTS_FLOAT32 (off by default) to lower trainer-vs-sampler logprob mismatch (#2673)
  • P/D and offload: Raiden decode replicas with TP4 (#2339); asynchronous Raiden decode load failures go through vLLM's native failed_recving path, with opt-in prefill send-completion signaling via additional_config.raiden_pd_send_completion (#2302); TPU_RAIDEN_TRANSFER_STREAMS sizes the TCP streams per push independently of the transfer rank count (#1911)
  • Startup validation and diagnostics: prefix caching on attention-free models is rejected at startup (#2597); set environment variables that nothing reads are reported at startup with a suggested name, and valid VLLM_TPU_* knobs no longer fail --fail-on-environ-validation (#2323)
  • Experimental AOT export: MTP draft export, verify windows, debug taps, KV paging at the attention kernel's block size, Gemma 4 support, and the model's own forward / compute_logits captured (#2709); KV cache groups for mixed sliding-window and full-attention models, a generic drafter name, the model id recorded as model_name, and the serving checkpoint written as Zarr v2 arrays in OCDBT (#2700)

Groundwork (not yet user-facing in this release)

  • Megakernel framework: SPMD helpers, PeerTopology and RemoteCopyBatch (#2363); ops/ and dma/ packages with lazy init (#2712); runtime contract and step statistics (#2713); plugin-facing runner views and hooks (#2715); VLLM_TPU_MEGAKERNEL config parsing and plugin fingerprint (#2716); bucket policy and step eligibility (#2908); rank-uniform decline agreement and startup guards (#2806); StepRouting per-step inspection and routing (#2948); pool layout updates with FP32 conv-state support (#2788); core spec, planner and profile registries with lazy builtins (#2797); megakernel/ moved to megakernel/core/ and megakernel_runtime/ to megakernel/adapter/ (#3019)
  • Model directory scaffolds under the Experimental tier: Kimi-K3 / Kimi-K3.1-Pro, home of the megakernel migration (#2837); MiniMax-M3.1-Flash (#2767)

Performance and kernel improvements

Attention / MLA

  • 128-token KV pages for dense FLASH_ATTN models with max_model_len > 8192, 8x fewer Raiden offload transfers (+47.3% tok/s on Qwen3-32B TP=8 8k/1k c=64; +56.2% on Llama-3.1-8B TP=8 at 131K context) (#2425)
  • Sparse MLA packs the FP8 RoPE cache to 64 B/token by default, 576 instead of 640 B/token: 10.4% more tokens in the same HBM on GLM-5.2, +2.7% to +4.0% TPS/chip at c=48 and c=64 (#2206)
  • PCP8 streaming RPA ring uses 3-slot KV buffering: kernel latency -8.7% to -15.6% on v7x, +1.8% to +5.5% tok/s end to end on PP4 x PCP8 (#2263)
  • TPU_MLA_SPARSE_SPLIT_Q is honored in non-DCP sparse MLA, where it was silently ignored (#2524)

MoE

  • Fused MoE v2 decode, parts 1 and 2 of 5: the next expert's token rows are issued under the current tile's compute (#3074), and the short height rung is placed at the expected expert load (T64 292.6 to 272.9 us, bit-exact) (#3075). See Note.
  • MoE routing prep uses one counting sort for EP one-hot and non-EP TP paths up to 2,048 rows: 25% to 63% less routing-prep time per layer on v7x and v6e, bit-identical (#2604)
  • Qwen3.5 MoE attention/MoE TP collectives overlapped with reduce-scatter and all-gather on large prefill buckets, opt-in with TPU_ATTN_MOE_OVERLAP=True (#1492)

GDN / hybrid

  • Internal prefill checkpoints for GDN prefix caching in Qwen3.5 and Qwen3.8, on by default; TPU_GDN_PREFILL_CHECKPOINT=0 turns it off (#2169)
  • GDN groups no longer take scheduler blocks on the compact-mamba layout (Qwen3.8-2.4T admits 93 instead of 89 requests per engine from a 6,720-block pool); on by default, TPU_SKIP_MAMBA_SCHEDULER_BLOCKS=0 opts out (#2783)
  • Opt-in (k, n) layout for in_proj_qkvz, TPU_GDN_QKVZ_KERNEL_LAYOUT=1: decode 119.1 to 110.9 ms per step on a 69-GDN-layer per-block FP8 model at DP16 x TP2 (#2719)
  • GDN single-token decode fuses the Q/K projection: pea...
Read more

Release v0.32.0

Choose a tag to compare

@charlesli640 charlesli640 released this 03 Nov 02:04

Basic Info

  • Cut: 2026-10-29 (branch point c7f724b2d on main), 2026-10-30 (two cherry-picked release-pipeline fixes, #2522 and #2634).
  • Tag commit: 1e5cd9e95 (v0.32.0, releases/v0.32.0).
  • vLLM: v0.32.0 (was v0.29.0 at v0.29.0); vllm-omni 0.32.0rc1 (was 0.29.0rc1).
  • Compared to: releases/v0.29.0 -- 230 PRs (excluding 12 automated support-matrix and verified-commit-hash bot commits, and 8 PRs that already shipped in v0.29.0 as cherry-picks; see Note).

Highlights

  • vLLM 0.32.0 (#1850, #2183) -- the plugin moves from vLLM 0.29.0 through 0.31.0 to 0.32.0, with vllm-omni 0.32.0rc1 and transformers>=5.18.0. Hybrid models keep kernel block ids in split KV cache groups, which vLLM 0.32 no longer provides.
  • Qwen4Exp (Qwen3.8-Flash-Next) on TPU (#1903, #1926, #1974) -- a native TPU text model with hyper-connection residuals that loads the released FP8 checkpoint, including the n-gram PLE layer.
  • Kimi-K3 DSpark speculative decoding and P/D (#1711, #1712) -- DSpark drafting over pooled KDA state, then prefill/decode disaggregation for it (GSM8K 97.65%, acceptance length 4.41 on v7x-32).
  • Experimental AOT export and Rust serving runtime (#2260, #2277, #2380, #2445, #2518, #2449) -- python -m vllm_torchtpu.experimental.export writes a vLLM model out as TorchTPU StableHLO serving artifacts; python -m vllm_torchtpu.experimental.rust serve serves them with the Rust runtime behind the vllm serve command line.
  • Opt-in fused MoE decode kernels (#1678, #1763, #1930) -- one Pallas call per rank for the routed experts, shared expert and all-reduce at low concurrency; the GLM-5.2 TP variant cuts P90 TPOT by 14.7% on an AgentX replay.
  • DeepSeek-V4 performance (#2377, #2161, #1854, #1729, #1982, #2290) -- kernel tuning for HCA/CSA attention, the compressor and the indexer (about 4% on AgentX c=160 with DSv4 Pro), SparseCore top-k overlapped with TensorCore work, and attention weights sharded across chiplets.
  • SRPT prefill scheduler (#1924, #2252) -- an opt-in scheduler class that orders chunked prefill by shortest remaining work, so long prefills no longer block short or prefix-cached requests, with an opt-in attention work cap for data-parallel decode.
  • Diffusion (vLLM-Omni) speedups (#1432, #1452, #1448, #1450) -- a Pallas flash-attention backend is now the default for diffusion transformers, plus a fused q/k RMSNorm + RoPE kernel and SparseCore all-to-alls for Wan.
  • Published on PyPI and Docker Hub (#1618, #2204, #2522, #1721) -- vllm-torchtpu-core and vllm-torchtpu wheels with PyPI trusted publishing and PEP 740 attestations; building no longer needs private registry access.

New features

Model support

  • Qwen4Exp (Qwen3.8-Flash-Next): native TPU text model with hyper-connection residuals, GDN linear attention and Qwen3Next sparse MoE (#1903); loads the released FP8 checkpoint as a text model (#1926); n-gram PLE layer and its weights (#1974)
  • Qwen2-family GQA models at TP=8 when the query-head count does not divide by TP, by zero-padding Q heads per KV group (Fin-R1-7B on v7x-8) (#1739)
  • Kimi-K3 + DSpark: speculative decoding with pooled KDA state (#1711) and prefill/decode disaggregation (#1712)
  • DeepSeek-V4: static-q speculative verify pass for full-length verify windows (#2129); attention wq_b/wo_a/wo_b sharded across chiplets (#1982); the compressor writes the indexer K cache in SEQ_ALONG_LANE (#2225); the HCA compressor state is excluded from prefix caching and KV offload (#2389)
  • Qwen3.5 MTP: prefill-only MTP on a PPxPCP producer for disaggregated speculative decoding (#1889)
  • GLM P/D: TP/DCP prefill to TP/DCP decode KV transfer, each prefill DCP rank sending its pages to the matching decode rank (#2211)
  • NVFP4 checkpoints fall back to ModelOptLinearMethod (#1257)

Diffusion (vLLM-Omni)

  • Pallas flash-attention backend for diffusion transformers, now the default (TPU_FLASH_ATTN); SDPA stays selectable and is the fallback for causal, masked or non-bf16 calls (#1432)
  • TPU device profiler for diffusion workers (#1428)
  • Wan: fused q/k RMSNorm + RoPE Pallas kernel (#1452), pair-swap matmul RoPE (#1448) with 2-D token-major cos/sin tables for vllm-omni 0.32 (#2226), GELU fenced from the down-projection fusion (#1449), timestep materialized before the transformer (#1451)
  • Dense all-to-alls offloaded to SparseCore in diffusion workers (#1450)
  • LTX-2: split RoPE applied on the q/k layout with reused tables (#2298)
  • vllm-omni pinned to 0.31.0rc1 with obsolete TPU patches dropped (#1945), then 0.32.0rc1 (#2183); libgl added to the omni image for multi-rank Ulysses (#1939)

Capabilities

  • Experimental AOT export to TorchTPU StableHLO serving artifacts (#2260): tensor-parallel programs, Orbax weight checkpoints, hybrid (GDN) models and FP8 KV cache (#2277); one serving_config in metadata.json (#2380); serve CLI arguments, speculative config, sort-free samplers and more backbones (#2445); the XLA flags libtpu runs with are recorded, and weight loading and saving are traced (#2518)
  • Experimental Rust runtime launcher: python -m vllm_torchtpu.experimental.rust serve <model> [vllm serve args] exports once and execs rvllm serve (#2449)
  • SRPT prefill scheduler (--scheduler-cls vllm_torchtpu.core.experimental.srpt_prefill_scheduler.SrptPrefillScheduler) (#1924), with an opt-in attention work cap, VLLM_TPU_SRPT_ATTN_WORK_CAP (#2252)
  • CUDA-matching RMSNorm and RoPE numerics: a tpu vLLM IR op provider for rms_norm / fused_add_rms_norm and out-of-tree RoPE CustomOps that compute in FP32 as the CUDA kernels do, to lower trainer-vs-rollout KL in RL (#2312)
  • RL sampling precision: logprob normalizer computed in FP32 without a full-vocab log_softmax (#2024), logprob quantization fixed (#1745), Gumbel-Max noise drawn in FP32 (#1747)
  • Offload and P/D: Raiden offloading connector admitted under pipeline parallelism (#2156); GLM-5.2 SparseCore MLA caches admitted in Raiden reshard (#1978); DCP fine prefix hits kept with a CPU offloading connector (#2040); PCP/TP/DP producer KV blocks registered ahead in the prompt-completing step (#1813); least_loaded decode routing in the disagg toy proxy (#2195)
  • Speculative decoding: RejectionSampler accepts an optional draft distribution for standard speculative sampling (groundwork; no caller passes it yet) (#2439); DSpark warns instead of erroring on config mismatches (#2187); better DSpark first-pass warmup (#1979)
  • Startup validation: the platform rejects a per-step token budget that cannot fit the largest compile_sizes entry, and non-integer compile_sizes, instead of crashing on the first large step (#1737)
  • TPU_SPARSE_MLA_CHUNK_TOKENS sets the sparse MLA chunk loop size (#2035)

Groundwork (not yet user-facing in this release)

  • Multi-LoRA: startup check for unsupported LoRA settings, LoRA in bf16 (#2372, #2418); shrink/expand ops for mixed-adapter batches (#2358); TPU Punica wrapper (#2375); recompile-free swap test (#2374); dtype-mismatch message under torch.compile (#2462). See Note: --enable-lora is not wired up in v0.32.0.
  • TurboQuant KV cache [1/3]: 128-lane KV page layout, quantization and VREG dequantization primitives; no existing code path uses them yet (#2228)
  • Megakernel framework (megakernel/): error hierarchy and numeric primitives (#2209), VMEM pool aliasing (#2227), execution config and default tiles (#2276), raw-pool layout and paged cache config (#2286), declarative model spec (#2292), raw-pool packing utilities (#2305), module options and declarations (#2318), pool backend factory (#2324), Pallas pool recurrent-state gather (#2361) and KV/checkpoint scatter (#2382) kernels, Block renamed to Module (#2390), execution profile registry (#2409), declaration-driven planner (#2448), PoolLayout derived from module cache declarations (#2476); fixes for redundant BF16 truncations (#2241) and unknown tunables / unvalidated chips (#2284)

Performance and kernel improvements

Attention / MLA

  • Batched RPA: seq-on-lane 128-byte aligned and a triple-buffered pipeline with in-step KV writeback (+0.6% to +1.3% throughput on Qwen3.5-397B-A17B-FP8, v7x-8) (#1420)
  • v2 ragged paged attention returns the LSE for MLA (#1639)
  • MLA DCP q-replicate uses a rank-independent dead q shard, so precompile peer replay works (#2190)

MoE

  • Fused MoE decode kernel for low concurrency, opt-in with FUSED_MOE_DECODE_KERNEL=1 (#1678)
  • GLM-5.2 fused TP decode MoE kernel with in-kernel routing, shared expert and all-reduce, opt-in with USE_MOE_TP_DECODE_KERNEL=1; 36% to 60% less time per decode MoE layer than EP16 on v7x TP16 (#1763)
  • Fused EP MoE: device-id ordered activation all-gather under DCP and EP, opt-in with MOE_FUSED_EP_SORTED_GATHER (#1930); uniform_random routing simulation honoured instead of disengaging (#1505); fp4 scale tables staged as fresh base tensors (#1857)

GDN / hybrid

  • GDN mixed and decode tile sizes unified and tuned to the memory budget (#1563)
  • Unified-pool GDN refactor: dense and pooled state share one granule-addressed path; pooled conv state defaults to FP32 (#1620)

DeepSeek-V4

  • HCA/CSA attention, compressor and indexer kernel tuning, bit-exact, about 4% on AgentX c=160 (#2377)
  • Indexer SparseCore top-k overlapped with TensorCore work (#2161)
  • Kernel block sizes tuned for long-context / AgentX workloads (#1854)
  • Transposed compressor weights materialized once instead of cloned every step (#1729)
  • StreamIndex: pass schedules shared across indexer layers and built in O(segments + steps) (#2290); small-bucket MIXED-pass blo...
Read more