Skip to content

v0.28.0

Latest

Choose a tag to compare

@yeahdongcn yeahdongcn released this 03 Nov 12:49
· 1 commit to v0.28.0-dev since this release
070c0ea

vLLM-MUSA v0.28.0 Release Notes

https://docs.mthreads.com/vllm-musa/vllm-musa-doc-online/S5000/releasenote/

中文版

概览

vLLM-MUSA v0.28.0 对齐上游 vLLM 0.28.0(pin 2cf0a69),是 v0.24.0 之后的下一个发布版本;相对 v0.24.0 分支(2733b6f),v0.28.0-dev 包含 70 个提交、9 位贡献者。本版本把引擎基线推进到 v0.28,新增 NVIDIA-Nemotron-3.5-Lightning-30B-A3B、GLM-5.3、DeepSeek-V4-Flash-0731 以及四款 OCR 模型的 S5000 服务支持,补齐 AWQ/GPTQ W4A16 量化、Mooncake 1P1D 分离式服务与混合 KV cache 快速路径,并把软件栈升级到 PyTorch 2.11.0.post2 / MATE 0.2.7 / torchada 0.1.95。

最终发布镜像

registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0

主要更新

上游基线

  • 同步上游 vLLM 0.28.0,适配新的 attention、fused-MoE、sampler、MLA、GDN 与 Model Runner 接口;vllm-musa 包版本同步更新为 0.1.28。
  • Model Runner 选择:MUSA 默认仍使用 Model Runner V1(DeepSeek-V4 的 FULL_DECODE_ONLY 路径更快),Model Runner V2 保留为显式 opt-in,DSpark 场景另有 V2 override。

新模型支持

  • NVIDIA-Nemotron-3.5-Lightning-30B-A3B(hybrid Mamba-2 / attention MoE):提供 TP1 + MTP6 的单请求低延迟配置,以及 TP2 + MTP4、最多 98 并发的吞吐配置,配套 mamba2 SSD prefill tile。 (#234, #238, #258)
  • GLM-5.3 FP8 / GLM-5.2:恢复 GLM MLA 路径,支持 TP8 + PP2 + EP 的 16 卡部署与 FULL_DECODE_ONLY graph 捕获;PP > 1 时 MCCL 改为 eager 初始化;GLM DSA sparse-MLA 的 MATE prefill 仅在 64 head / 576 width / index_topk=2048 形态下启用,其余形态回落到通用 TileLang 稀疏路径。 (#236)
  • DeepSeek-V4-Flash-0731:启用 DSpark4 推测解码与 learned indexer,新增单机与 1P1D 两种配置。 (#239, #269)
  • OCR 模型族:DeepSeek-OCR、MinerU2.5-Pro-2605-1.2B、OvisOCR2、PaddleOCR-VL-1.6 均提供 1× S5000 / TP1 的 BF16 配置;视觉 rotary 快路径统一由 optimization contract 门控,DeepSeek-OCR 的视觉 attention 默认走 Triton,并修正其 MUSA 精度问题。 (#212, #266, #267, #268, #315)
  • Qwen:新增 Qwen3.5-122B-A10B(TP8 + MTP3)、Qwen3-Next-80B-A3B-Instruct(TP8,piecewise graph)、Qwen3.8-27B(TP2 + MTP3)与 Qwen3.5 小模型(0.8B/2B/4B/9B,MTP1)配置;多模态侧加入 Qwen3-VL MM encoder 的 FlashAttention 路由与可选的视觉块 graph capture。
  • 模型专用镜像:随本发布线提供 qwen38-flash-next、glm53-flash、hy4-preview(vllm-openai)与 flux3(vllm-omni)等 tag。这些 tag 独立于本分支维护,对应模型未在本分支内验证,使用时请自行核对镜像与模型 revision。

注意力与 KV cache

  • attention backend 能力声明:各 backend 必须重新声明上游的 supports_* 能力,新增能力一致性测试;MUSAFlashAttentionBackend 补齐 supports_sliding_window,使 interleaved sliding/full 模型可跑在 FLASH_ATTN 上。 (#254)
  • diffusion 固定 TRITON_ATTN:diffusion 模型的 encoder/denoise 请求需要逐请求 causal 掩码,而 MATE FA3 的 causal 是 bool、MUSA 又没有 FA4 的 dynamic_causal,所以 FLASH_ATTN 遇到 tensor 形式的 causal 会直接报错;diffusion 模型即使显式指定 --attention-backend FLASH_ATTN 也会被覆盖并告警。 (#254)
  • 混合 attention page-size 快路径:默认固定 64-token block 并输出日志,align 模式的 Mamba block table 由单个 Triton kernel 生成;显式 --block-size 不再被静默覆盖,而是给出告警并关闭该快路径。 (#285)
  • Mamba / 混合 KV:修复 prefix-cache pool 归属、淘汰与 headroom gating,Mooncake 场景使用专用 pool;另补 GDN 全量 decode metadata 复用与 uniform-decode 下跳过 SSM slot-mapping(patch 0101、0103)。 (#221, #226, #235, #247)
  • MLA:prefill 由 MUSAMLAPrefillBackend 接管(不再走 MLACommonImpl 的报错分支),chunked-context prefill 迁移到 vLLM v0.28 的 per-request chunk layout;MUSA 上关闭 _pad_v,output_scale 仍未实现。 (#260)
  • DeepSeek-V4:新增原生 512-d sparse compressor(含 save_partial 合并写入)与 FLASHMLA Q-head pad 融合;稀疏 indexer top-k 使用独立 MUSA op;长 prefill indexer logits 以 512 MiB 预算和 512 行 tile 上限约束;learned indexer 保留在 graph decode;aux-stream overlap 仅在 FULL graph 中启用。 (#245, #253, #283)

量化与 MoE

  • AWQ/GPTQ W4A16:在 MUSA 上改用 Triton W4A16 kernel(沿 AMD/ROCm 路线,不引入 Marlin);M > 64 时约束到 BLOCK_M=128 / BLOCK_N=64 / BLOCK_K=32(MUSA 上 BLOCK_N=128、BLOCK_K>=64 会返回 NaN 且跨次运行不一致),M ≤ 64 与上游逐字节一致。 (#277)
  • W4A16 decode 优化:split-K(2–8 段、fp32 累加、每段不跨量化组)、torch.library.custom_op 包装(避免 Dynamo 在 trace 时冻结按 M 选 tile 的分支)、gate_up 改用 16×64 tile(BLOCK_K=64)。 (patch 0185–0188)
  • TileLang grouped top-k router:新增 grouped softmax top-k 路由 kernel 的 TileLang 实现(最多 512 experts / 128 groups / top-k 32),带独立 kernel 测试。 (#192)
  • Router gate:新增 router_gate_fp32,2-D grid FP32 GEMM 取代 grid=(1,1,1) 的 vendor SGEMM,保持 FMA 累加顺序(bitwise 一致),并提供 VLLM_MUSA_ROUTER_GATE_FP32=0 关闭开关。 (#241)
  • tiny-M MoE:M ≤ 32 的默认配置从 4 个 pipeline stage 改为 1 个。 (#242)
  • DeepGEMM:适配 MATE 0.2.6+ 的 grouped wrapper。 (#207)

推测解码

  • DSpark 与 MTP 组合:新增 DSpark(DeepSeek-V4-Flash-0731)与 Nemotron MTP6,并覆盖 Qwen3.5 MTP1、Qwen3.6 MTP2/MTP3、Qwen3.8 MTP3。
  • Model Runner V2 sampler:rejection sampler kernel 适配 MUSA Triton 的 scalar predicate 与 Gumbel helper 约定,不改变上游接受/重采样算法。
  • async MTP:修复 metadata 归属与 state 就绪事件。

性能

以下数字取自相关提交的测量记录(本发布流程未重新跑测,也不是 v0.24.0 → v0.28.0 的整体基准;除注明外为 S5000):

变更 测量结果 条件
W4A16 decode (#277, patch 0185/0187/0188) M=1 时 o_proj/down_proj 只暴露 24–64 个 program、带宽 7–13 GB/s(峰值 1570 GB/s),split-K 拆 2–8 段以抬高 program 数;gate_up 的 decode tile 由 32×32 改为 16×64(BLOCK_K=64),带宽 100 → 240 GB/s S5000 kernel 级测量,数字取自 patch 0185/0188 的提交说明
Router gate FP32 并行 GEMM (#241) kernel 1.77×/1.92×/2.04×(M=1/7/115),M=512 为 1.02×,bitwise 一致;E2E TPOT 8.1576 → 7.49 ms/token(-8.2%),总时长 9.5808 → 8.9134 s(-7.0%),TTFT 1.4294 → 1.4321 s(持平),输出 SHA 一致 Nemotron-3.5 MTP6,TP1,batch 1,4096-in/1000-out,1 warmup + 3 次测量
tiny-M MoE 单 stage (#242, patch 0168) 11.635 → 9.563 ms/token(-17.80%),输出逐字节一致(9 个请求同一 SHA),TTFT 基本持平(1648 vs 1678 ms) S5000,4096-in/128-out,单请求,Nemotron-3.5 MTP6 decode 形状
mamba2 SSD prefill tile 8 个 warp (#258) 单次调用 1476 → 756 µs,TTFT 1028.03 → 898.55 ms(-12.59%) chunk 128 / hdim 64 / dstate 128 / groups ratio 8

此外还有一批未附具体倍数的性能改动:DeepSeek-V4 长上下文高并发 prefill、1P1D 下的 prefill/decode 调优(#269, #275)、交错 MRoPE 改为 strided copy 重建(#246)、Model Runner V2 按 in-flight step 轮转 output copy stream(#263)、capability 检查不再重复初始化 MTML(#278)。

分离式服务(PD / Mooncake)

  • Mooncake wheel:升级到 0.3.13。
  • K/V-first cache layout:Mooncake 接受 MUSA FlashAttention 的 K/V-first 布局(独立 dense K/V 区域,或 padded-page-aware 的 blocks-first 混合视图),不改变 kernel 侧 cache 格式。 (#215)
  • KV load 失败上报:失败会回报给上层,不再静默丢失。 (#276)
  • 示例:新增 example/disaggregated_serving/,含 Mooncake store offload 与 1P1D(MooncakeConnector + MooncakeStoreConnector)可直接运行的脚本与前置条件说明。 (#264)

构建、补丁系列与镜像

  • 补丁规模:构建期 patch series 由 122 个增至 194 个,并新增 patches/series/upstream/(3 个 upstream cherry-pick,上游 pin 升级后自动剔除已合入的补丁)。
  • 补丁系列:改为 git am -3 可回放,保留 index blob 行、连续编号,并由 tools/musa_sync.py check-series --replay/--round-trip 校验 mailbox 形状、真实回放与 regen 字节不动点;module-drift tripwire 覆盖到 MLA common 模块。 (#250)
  • patch 0138:重新生成(Mamba prefix-cache pool 归属)。 (#289)
  • Docker 构建:vendored vLLM 依赖安装加 --constraint requirements/common.txt;安装后审计覆盖完整 MUSA cohort(mate、mate-mubin、flash_attn_3、flash_mla、deep-gemm、flashinfer-python、sageattention、tilelang_musa、apache-tvm-ffi、torch-c-dlpack-ext)。
  • 镜像元数据:新增 OCI label com.mthreads.vllm-musa.mooncake-version;tag flavor 更新为 ..._pytorch_release_2.11.0.post2_musa5.2.0。
  • 启动路径:custom op 在 AOT cache load 之前预加载;CUDA graph capture 前后正确处理 expandable segments,并保留其余 allocator 设置;MTML 生命周期与 vLLM v0.28 对齐。 (#284, #291, #298)

文档与示例

  • Cookbook:扩展到 Qwen / DeepSeek / GLM / Nemotron / OCR 五个族,每行统一标注精度、硬件、推测解码与 recipe。
  • 示例:新增 example/generate/(Qwen3.5-35B-A3B-FP8 离线与在线示例);docs/example/ 由 example/disaggregated_serving/ 取代(README 重写,并加入 Mooncake store 与 1P1D 的可运行脚本)。
  • 安装文档:补充双 index 安装方式;mineru-vl-utils 纳入运行时依赖。

Cookbook 配置

以下配置来自 v0.28.0 serving cookbook,面向 S5000、约 4K 输入 / 1K 输出的有界并发场景。请把同一行的精度、TP 和推测解码设置作为一个整体复制;更换模型、并发或上下文长度后需要重新校准。

默认镜像:registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0

Qwen

模型 精度 硬件 / 并行 推测解码
Qwen3-8B FP8 1× S5000 / TP1 Off
Qwen3.5-0.8B BF16 1× S5000 / TP1 MTP1
Qwen3.5-2B BF16 1× S5000 / TP1 MTP1
Qwen3.5-4B BF16 1× S5000 / TP1 MTP1
Qwen3.5-9B BF16 2× S5000 / TP2 MTP1
Qwen3.5-27B BF16 / FP8 2× S5000 / TP2 MTP1
Qwen3.5-35B-A3B BF16 / FP8 4× S5000 / TP4 MTP3
Qwen3.5-122B-A10B BF16 / FP8 8× S5000 / TP8 MTP3
Qwen3.5-397B-A17B FP8 8× S5000 / TP8 Off
Qwen3.6-27B BF16 / FP8 2× S5000 / TP2 MTP3
Qwen3.6-35B-A3B BF16 / FP8 4× S5000 / TP4 MTP2
Qwen3.8-27B BF16 / FP8 2× S5000 / TP2 MTP3
Qwen3-Next-80B-A3B-Instruct BF16 8× S5000 / TP8 Off
Qwen3-VL-8B-Instruct BF16 / FP8 1× S5000 / TP1 Off
Qwen3-VL-30B-A3B-Instruct BF16 / FP8 4× S5000 / TP4 Off
Qwen3-VL-32B-Instruct BF16 / FP8 4× S5000 / TP4 Off

DeepSeek / GLM / Nemotron / OCR

模型 精度 硬件 / 并行 推测解码
DeepSeek-V4-Flash FP8 KV cache 8× S5000 / TP8 Fixed MTP4(MTP-off 可选)
DeepSeek-V4-Flash-0731 FP8 8× S5000 / TP8(1P1D 时为每节点 8 卡) DSpark4(可选 1P1D + Mooncake store)
GLM-5.3 / GLM-5.2 FP8 16× S5000 / TP8 + PP2(EP 开启,FULL_DECODE_ONLY) —
NVIDIA-Nemotron-3.5-Lightning-30B-A3B BF16 1× S5000 / TP1 MTP6
NVIDIA-Nemotron-3.5-Lightning-30B-A3B BF16 2× S5000 / TP2 MTP4(最多 98 并发)
DeepSeek-OCR BF16 1× S5000 / TP1 —
MinerU2.5-Pro-2605-1.2B BF16 1× S5000 / TP1 —
OvisOCR2 BF16 1× S5000 / TP1 —
PaddleOCR-VL-1.6 BF16 1× S5000 / TP1 —

表中 Off 表示 cookbook 明确关闭推测解码,— 表示该 recipe 未涉及。Nemotron-3.5-Lightning 的两个配置是延迟与吞吐的取舍:TP1(--max-num-seqs 1,MTP6)面向单请求的逐 token 延迟,需要并发请改用 TP2(MTP4,最多 98 并发);两个配置的启动参数应整体复制。

软件栈

组件 v0.24.0 v0.28.0
vLLM(上游 pin) 0.24.0 (ee0da84) 0.28.0 (2cf0a69)
vllm-musa 包 0.1.24 0.1.28
Python 3.10 3.10
PyTorch / torch_musa 2.11.0.post1+musa5.2.0 2.11.0.post2+musa5.2.0
MATE 0.2.4 0.2.7
flash_attn_3 / flash_mla / deep-gemm / sageattention 0.2.4 0.2.7+musa
mate-mubin — 0.2.7
flashinfer-python — 0.2.7+musa
torchada 0.1.83 0.1.95
deep_ep 1.1.0+musa5.2.0torch2.11.0.post1 1.1.0+musa5.2.0torch2.11.0.post2
tilelang_musa 0.1.8+musa.3 0.1.12+musa.2
apache-tvm-ffi 0.1.9.post3+musa.1 0.1.11.post1+musa.1
Triton 3.2.0 3.2.0
Transformers 5.5.3 5.15.0
mthreads-ml-py >=2.4.2 >=2.4.3
Mooncake MUSA wheel 0.3.12.post1 0.3.13

Bug 修复

  • 修复 AR+RMSNorm 重写插入节点缺失 fake-tensor meta 的问题。 (#261)
  • 修复 Mooncake 未上报失败 KV load 的问题。 (#276)
  • 修复 Qwen3 fused op 参数传递与嵌套 multimodal config 缺 architectures 的兼容问题。 (#232, #237)
  • 修复无 GPU 构建环境下的 TileLang import 问题,并收窄其 workaround 范围。 (#214, #219)
  • 修复 MinerU 视觉语言运行时依赖缺失(mineru-vl-utils)。 (#313)

升级说明

推荐升级方式:直接更换镜像,不要手动升级依赖。

v0.28.0 是一次跨越式升级:torch/torch_musa 到 2.11.0.post2、MATE 系列包到 0.2.7、torchada 到 0.1.95、tilelang_musa 到 0.1.12、transformers 到 5.15.0。这些包之间有版本耦合(MATE 与 flash_attn_3/flash_mla/deep-gemm/flashinfer-python 同批次发布,tilelang_musa 绑定 apache-tvm-ffi),镜像构建时会对整个 cohort 做一次版本审计,因此换镜像是唯一省事且不会漏包的路径:

docker pull registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0
  • 不建议在旧镜像里 pip install -U vllm-musa:这样只会升插件包,torch_musa、MATE、torchada、tilelang_musa 仍是旧版本,容易装出官方未验证的混搭组合。
  • 必须从源码构建时,请按 docs/installation.md 的双 index 顺序整体安装(先 musa_private.txt + --no-deps,再 build.txt/common.txt,最后 pip install . --no-build-isolation),不要逐个升级包。
  • 版本标识:上游 vLLM 0.28.0(pin 2cf0a69),vllm-musa 插件包 metadata 版本 0.1.28。
  • 设备可见性:由部署环境显式设置 MUSA_VISIBLE_DEVICES,镜像不预置固定设备列表。
  • 运行时判断:MUSA 环境不要用 torch.cuda.is_available(),应使用项目提供的 MUSA platform helper。

已知限制

  • W4A16 tile 约束:M > 64 时 Triton W4A16 固定 BLOCK_M=128 / BLOCK_N=64 / BLOCK_K=32;BLOCK_N=128 或 BLOCK_K>=64 的 tile 在 MUSA 上会返回 NaN,且跨次运行不一致。
  • 混合 attention 的 64-token 快路径与显式 --block-size 互斥,显式指定时会告警并关闭快路径。
  • FLASH_ATTN 不支持逐请求 causal:MUSA 没有 FA4 的 dynamic_causal,MATE FA3 的 causal 是 bool,因此遇到 tensor causal 会直接报错(reject_per_sequence_causal)。diffusion 模型(model_config.is_diffusion)的 encoder/denoise 请求需要逐请求 causal,因此被固定到 TRITON_ATTN;显式指定 --attention-backend FLASH_ATTN 会被覆盖并告警,其他显式 backend 保持原样。 (#254)
  • MLA:MUSA 的 MLA prefill 固定走 MATE FlashAttention;output_scale 仍未实现。GLM DSA 的 MATE sparse prefill 仅覆盖 64 head / 576 width / index_topk=2048 形态。
  • MinerU 依赖:engine 与 client 两侧都需要 mineru-vl-utils 才能运行其 logits processor;普通 chat completion 不会触发该 pipeline。
  • Model Runner:MUSA 与 DeepSeek-V4 默认使用 Model Runner V1(本版本未改变);DSpark 场景使用 V2 override,其他情况需要 V2 请显式指定并按模型重新验证 graph 捕获。
  • Cookbook 参数:面向约 4K 输入 / 1K 输出的有界并发场景;其他请求形状需要重新调整 max-num-batched-tokens、max-num-seqs 和 graph capture 配置。
  • GLM-5.3 的 no-thinking 行为依赖 cookbook 中的模板 workaround;GLM-5.2 使用 checkpoint 自带模板。
  • DeepSeek-V4-Flash-0731 的 1P1D 配置需要 RoCE、Mooncake MUSA wheel、SSD offload 目录、约 400 GiB 空闲 host DRAM(128K 配置)与 ulimit -l unlimited。
  • FP8、MTP 和 CUDA Graph 路径对模型 revision、依赖版本和输入 shape 敏感,生产部署前应完成真实请求与语义检查。

English Version

Overview

vLLM-MUSA v0.28.0 aligns with upstream vLLM 0.28.0 (pinned at 2cf0a69) and is the next release after v0.24.0; against the v0.24.0 branch (2733b6f), v0.28.0-dev carries 70 commits from 9 contributors. It moves the engine baseline to v0.28, adds S5000 serving support for NVIDIA-Nemotron-3.5-Lightning-30B-A3B, GLM-5.3, DeepSeek-V4-Flash-0731 and four OCR models, enables AWQ/GPTQ W4A16 quantization, Mooncake 1P1D disaggregated serving and a mixed-pool KV cache fast path, and upgrades the stack to PyTorch 2.11.0.post2 / MATE 0.2.7 / torchada 0.1.95.

Final Release Image

registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0

Highlights

Upstream alignment

  • Synchronized with upstream vLLM 0.28.0, adapting the attention, fused-MoE, sampler, MLA, GDN and Model Runner interfaces; the vllm-musa package version is now 0.1.28.
  • Model Runner selection: MUSA still defaults to Model Runner V1 (the DeepSeek-V4 FULL_DECODE_ONLY path is faster there), with Model Runner V2 kept as an explicit opt-in and a separate V2 override for DSpark.

New model support

  • NVIDIA-Nemotron-3.5-Lightning-30B-A3B (hybrid Mamba-2 / attention MoE): a TP1 + MTP6 profile tuned for single-request decode latency and a TP2 + MTP4 profile admitting up to 98 concurrent sequences, backed by mamba2 SSD prefill tiles. (#234, #238, #258)
  • GLM-5.3 FP8 / GLM-5.2: restores the GLM MLA path and supports 16-GPU TP8 + PP2 + EP deployments with FULL_DECODE_ONLY graph capture; MCCL init is eager when PP > 1; GLM DSA sparse-MLA MATE prefill is enabled only for the 64-head / 576-width / index_topk=2048 shape, with everything else on the generic TileLang sparse route. (#236)
  • DeepSeek-V4-Flash-0731: enables DSpark4 speculative decoding with the learned indexer, in single-node and 1P1D shapes. (#239, #269)
  • OCR family: DeepSeek-OCR, MinerU2.5-Pro-2605-1.2B, OvisOCR2 and PaddleOCR-VL-1.6 each ship a 1× S5000 / TP1 BF16 profile; vision rotary fast paths are gated by the optimization contract, DeepSeek-OCR vision attention defaults to Triton, and its MUSA precision issue is fixed. (#212, #266, #267, #268, #315)
  • Qwen: new Qwen3.5-122B-A10B (TP8 + MTP3), Qwen3-Next-80B-A3B-Instruct (TP8, piecewise graphs), Qwen3.8-27B (TP2 + MTP3) and small Qwen3.5 models (0.8B/2B/4B/9B, MTP1) profiles; the multimodal side gains FlashAttention routing for Qwen3-VL MM encoders and optional vision-block graph capture.
  • Model-specific images: this release line also publishes tags qwen38-flash-next, glm53-flash, hy4-preview (vllm-openai) and flux3 (vllm-omni). These tags are maintained independently of this branch and their models are not validated in it; verify both the image and the model revision yourself.

Attention and KV cache

  • Attention backend capabilities: every backend must re-declare the upstream supports_* capabilities, backed by a new capability-parity test; MUSAFlashAttentionBackend gained supports_sliding_window, so interleaved sliding/full models can run on FLASH_ATTN. (#254)
  • Diffusion is pinned to TRITON_ATTN: diffusion encoder/denoise requests need per-request causal masks, but MATE FA3 types causal as a bool and MUSA has no FA4 dynamic_causal, so FLASH_ATTN raises on a tensor causal; an explicit --attention-backend FLASH_ATTN is overridden with a warning. (#254)
  • Hybrid attention page-size fast path: the 64-token block is pinned by default and logged, and align-mode Mamba block tables are built by a single Triton kernel; an explicit --block-size is no longer silently overridden — it warns and disables the fast path. (#285)
  • Mamba / hybrid KV: fixed prefix-cache pool ownership, eviction and headroom gating, with a dedicated pool for Mooncake, plus common GDN full-decode metadata reuse and SSM slot-mapping elision in uniform decode (patches 0101, 0103). (#221, #226, #235, #247)
  • MLA: prefill is now owned by MUSAMLAPrefillBackend (no longer the MLACommonImpl error branch), and chunked-context prefill migrated to the vLLM v0.28 per-request chunk layout; _pad_v is disabled on MUSA and output_scale remains unimplemented. (#260)
  • DeepSeek-V4: native 512-d sparse compressor (folding in the save_partial state write) and FLASHMLA Q-head pad fusion; sparse indexer top-k runs on a dedicated MUSA op; long-prefill indexer logits are bounded by a 512 MiB budget and a 512-row tile cap; the learned indexer is retained on graph decode; aux-stream overlap is confined to FULL graphs. (#245, #253, #283)

Quantization and MoE

  • AWQ/GPTQ W4A16 on MUSA now uses the Triton W4A16 kernel (the AMD/ROCm route, no Marlin port); for M > 64 the tile is constrained to BLOCK_M=128 / BLOCK_N=64 / BLOCK_K=32 (BLOCK_N=128 and BLOCK_K>=64 return NaN and are not run-to-run deterministic on MUSA), while M ≤ 64 stays byte-identical to upstream. (#277)
  • W4A16 decode work: split-K (2–8 pieces, fp32 reduction, no piece crossing a quantization group), a torch.library.custom_op wrapper so Dynamo cannot freeze the per-M tile branch at trace time, and a 16×64 tile with BLOCK_K=64 for gate_up. (patches 0185–0188)
  • TileLang grouped top-k router: a TileLang implementation of the grouped softmax top-k route (up to 512 experts / 128 groups / top-k 32), with its own kernel tests. (#192)
  • Router gate: the new router_gate_fp32 2-D-grid FP32 GEMM replaces the grid=(1,1,1) vendor SGEMM while preserving FMA accumulation order (bitwise identical), with a VLLM_MUSA_ROUTER_GATE_FP32=0 kill switch. (#241)
  • tiny-M MoE: the default for M ≤ 32 moves from 4 pipeline stages to 1. (#242)
  • DeepGEMM: adapted the grouped wrappers to MATE 0.2.6+. (#207)

Speculative decoding

  • DSpark and MTP coverage: added DSpark (DeepSeek-V4-Flash-0731) and Nemotron MTP6, and covered Qwen3.5 MTP1, Qwen3.6 MTP2/MTP3 and Qwen3.8 MTP3.
  • Model Runner V2 sampler: rejection-sampler kernels were adapted to the MUSA Triton scalar-predicate and Gumbel-helper contracts without changing upstream acceptance or resampling.
  • async MTP: fixed metadata ownership and state-readiness events.

Performance

Numbers below are taken from the cited commits (not rerun for this release, and not a v0.24.0 → v0.28.0 benchmark; S5000 unless noted):

Change Measured Conditions
W4A16 decode (#277, patches 0185/0187/0188) At M=1, o_proj/down_proj expose only 24–64 programs and stream at 7–13 GB/s against a 1570 GB/s peak, so split-K breaks K into 2–8 pieces; the gate_up decode tile moves from 32×32 to 16×64 with BLOCK_K=64, lifting bandwidth from 100 to 240 GB/s S5000 kernel-level measurements quoted from the patch 0185/0188 commit messages
Parallel FP32 router-gate GEMM (#241) Kernel 1.77×/1.92×/2.04× (M=1/7/115), 1.02× at M=512, bitwise identical; end-to-end TPOT 8.1576 → 7.49 ms/token (-8.2%), total time 9.5808 → 8.9134 s (-7.0%), TTFT 1.4294 → 1.4321 s (unchanged), identical output SHA Nemotron-3.5 MTP6, TP1, batch 1, 4096-in/1000-out, 1 warmup + 3 measured runs
One pipeline stage for tiny-M MoE (#242, patch 0168) 11.635 → 9.563 ms/token (-17.80%), byte-identical output (same SHA across 9 requests), TTFT unchanged (1648 vs 1678 ms) S5000, 4096-in/128-out, one request, Nemotron-3.5 MTP6 decode shape
mamba2 SSD prefill tile at eight warps (#258) 1476 → 756 µs per call, TTFT 1028.03 → 898.55 ms (-12.59%) chunk 128 / hdim 64 / dstate 128 / groups ratio 8

A further set of performance changes ships without published multipliers: DeepSeek-V4 long-context high-concurrency prefill and 1P1D prefill/decode tuning (#269, #275), interleaved MRoPE rebuilt with a strided copy (#246), Model Runner V2 rotating its output copy stream per in-flight step (#263), and capability checks that no longer reinitialize MTML (#278).

Disaggregated serving (PD / Mooncake)

  • Mooncake wheel: upgraded to 0.3.13.
  • K/V-first cache layout: Mooncake accepts MUSA FlashAttention's K/V-first cache layout (separate dense K/V regions, or a padded-page-aware blocks-first hybrid view) without changing the kernel-facing cache format. (#215)
  • KV load failures: now reported upward instead of being dropped silently. (#276)
  • Examples: new example/disaggregated_serving/ with runnable Mooncake store-offload and 1P1D (MooncakeConnector + MooncakeStoreConnector) scripts with their prerequisites. (#264)

Build, patch series and image

  • Patch volume: the build-time series grows from 122 to 194 patches, plus a new patches/series/upstream/ (3 upstream cherry-picks, auto-dropped once the upstream pin already contains them).
  • Patch series: now git am -3 replayable — index blob lines are preserved, numbering stays contiguous, and tools/musa_sync.py check-series --replay/--round-trip validates mailbox shape, a real replay and the regen byte fixed point; module-drift tripwires now cover the MLA common module. (#250)
  • Patch 0138: regenerated (Mamba prefix-cache pool ownership). (#289)
  • Docker build: vendored vLLM dependencies install with --constraint requirements/common.txt, and the post-install audit covers the full MUSA cohort (mate, mate-mubin, flash_attn_3, flash_mla, deep-gemm, flashinfer-python, sageattention, tilelang_musa, apache-tvm-ffi, torch-c-dlpack-ext).
  • Image metadata: new OCI label com.mthreads.vllm-musa.mooncake-version; the tag flavor is now ..._pytorch_release_2.11.0.post2_musa5.2.0.
  • Startup: custom ops are preloaded before the AOT cache load; expandable segments are handled around graph capture while the rest of the allocator configuration is preserved; MTML lifecycle handling is aligned with vLLM v0.28. (#284, #291, #298)

Documentation and examples

  • Cookbook: now spans five families — Qwen, DeepSeek, GLM, Nemotron and OCR — with precision, hardware, speculative decoding and a recipe for every row.
  • Examples: added example/generate/ (Qwen3.5-35B-A3B-FP8 offline and online examples); docs/example/ is replaced by example/disaggregated_serving/ (rewritten README plus runnable Mooncake store and 1P1D scripts).
  • Installation docs: cover the dual-index install; mineru-vl-utils is now a runtime dependency.

Documented Cookbook Profiles

The profiles below are from the v0.28.0 serving cookbook and target S5000 hardware, approximately 4K input / 1K output, and bounded concurrency. Copy the precision, TP and speculative-decoding settings of a row as a unit; changing the model, concurrency or context length requires recalibration.

Default image: registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0

Qwen

Model Precision Hardware / parallelism Speculative decoding
Qwen3-8B FP8 1× S5000 / TP1 Off
Qwen3.5-0.8B BF16 1× S5000 / TP1 MTP1
Qwen3.5-2B BF16 1× S5000 / TP1 MTP1
Qwen3.5-4B BF16 1× S5000 / TP1 MTP1
Qwen3.5-9B BF16 2× S5000 / TP2 MTP1
Qwen3.5-27B BF16 / FP8 2× S5000 / TP2 MTP1
Qwen3.5-35B-A3B BF16 / FP8 4× S5000 / TP4 MTP3
Qwen3.5-122B-A10B BF16 / FP8 8× S5000 / TP8 MTP3
Qwen3.5-397B-A17B FP8 8× S5000 / TP8 Off
Qwen3.6-27B BF16 / FP8 2× S5000 / TP2 MTP3
Qwen3.6-35B-A3B BF16 / FP8 4× S5000 / TP4 MTP2
Qwen3.8-27B BF16 / FP8 2× S5000 / TP2 MTP3
Qwen3-Next-80B-A3B-Instruct BF16 8× S5000 / TP8 Off
Qwen3-VL-8B-Instruct BF16 / FP8 1× S5000 / TP1 Off
Qwen3-VL-30B-A3B-Instruct BF16 / FP8 4× S5000 / TP4 Off
Qwen3-VL-32B-Instruct BF16 / FP8 4× S5000 / TP4 Off

DeepSeek / GLM / Nemotron / OCR

Model Precision Hardware / parallelism Speculative decoding
DeepSeek-V4-Flash FP8 KV cache 8× S5000 / TP8 Fixed MTP4 (MTP-off available)
DeepSeek-V4-Flash-0731 FP8 8× S5000 / TP8 (8 GPUs per node in 1P1D) DSpark4 (optional 1P1D + Mooncake store)
GLM-5.3 / GLM-5.2 FP8 16× S5000 / TP8 + PP2 (EP enabled, FULL_DECODE_ONLY) —
NVIDIA-Nemotron-3.5-Lightning-30B-A3B BF16 1× S5000 / TP1 MTP6
NVIDIA-Nemotron-3.5-Lightning-30B-A3B BF16 2× S5000 / TP2 MTP4 (up to 98 concurrent sequences)
DeepSeek-OCR BF16 1× S5000 / TP1 —
MinerU2.5-Pro-2605-1.2B BF16 1× S5000 / TP1 —
OvisOCR2 BF16 1× S5000 / TP1 —
PaddleOCR-VL-1.6 BF16 1× S5000 / TP1 —

In the tables, Off means the cookbook explicitly disables speculative decoding, while — means the recipe does not cover it. The two Nemotron-3.5-Lightning profiles trade throughput for latency: TP1 (--max-num-seqs 1, MTP6) targets per-token latency for a single request, so concurrent traffic should use TP2 (MTP4, up to 98 sequences). Copy each profile's launch arguments as a unit.

Software Stack

Component v0.24.0 v0.28.0
vLLM (upstream pin) 0.24.0 (ee0da84) 0.28.0 (2cf0a69)
vllm-musa package 0.1.24 0.1.28
Python 3.10 3.10
PyTorch / torch_musa 2.11.0.post1+musa5.2.0 2.11.0.post2+musa5.2.0
MATE 0.2.4 0.2.7
flash_attn_3 / flash_mla / deep-gemm / sageattention 0.2.4 0.2.7+musa
mate-mubin — 0.2.7
flashinfer-python — 0.2.7+musa
torchada 0.1.83 0.1.95
deep_ep 1.1.0+musa5.2.0torch2.11.0.post1 1.1.0+musa5.2.0torch2.11.0.post2
tilelang_musa 0.1.8+musa.3 0.1.12+musa.2
apache-tvm-ffi 0.1.9.post3+musa.1 0.1.11.post1+musa.1
Triton 3.2.0 3.2.0
Transformers 5.5.3 5.15.0
mthreads-ml-py >=2.4.2 >=2.4.3
Mooncake MUSA wheel 0.3.12.post1 0.3.13

Bug Fixes

  • Fixed missing fake-tensor meta on nodes inserted by the AR+RMSNorm rewrite. (#261)
  • Fixed Mooncake not reporting failed KV loads. (#276)
  • Fixed Qwen3 fused-op argument passing and the compatibility of nested multimodal configs without architectures. (#232, #237)
  • Fixed the TileLang import failure on GPU-less build hosts and narrowed the workaround. (#214, #219)
  • Fixed the missing MinerU vision-language runtime dependency (mineru-vl-utils). (#313)

Upgrade Notes

Upgrade by switching the image. Do not upgrade the dependencies by hand.

v0.28.0 is a step change: torch/torch_musa move to 2.11.0.post2, the MATE package family to 0.2.7, torchada to 0.1.95, tilelang_musa to 0.1.12 and transformers to 5.15.0. These packages are coupled (MATE ships in lockstep with flash_attn_3 / flash_mla / deep-gemm / flashinfer-python, and tilelang_musa pins apache-tvm-ffi), and the image build audits the whole cohort. Switching the image is the only path that is both simple and complete:

docker pull registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0
  • Do not run pip install -U vllm-musa inside an older image: that upgrades the plugin only, leaving torch_musa, MATE, torchada and tilelang_musa behind and producing a combination nobody validated.
  • If you must build from source, follow the dual-index order in docs/installation.md as a whole (musa_private.txt with --no-deps, then build.txt/common.txt, then pip install . --no-build-isolation) instead of upgrading packages one by one.
  • Version identifiers: upstream vLLM 0.28.0 (pin 2cf0a69); vllm-musa plugin metadata version 0.1.28.
  • Device visibility: set MUSA_VISIBLE_DEVICES explicitly in the deployment environment; the image does not bake in a fixed device list.
  • Runtime detection: do not use torch.cuda.is_available() to detect MUSA availability; use the project's MUSA platform helper.

Known Limitations

  • W4A16 tile constraint: for M > 64 the Triton W4A16 kernel is fixed to BLOCK_M=128 / BLOCK_N=64 / BLOCK_K=32; tiles with BLOCK_N=128 or BLOCK_K>=64 return NaN and are not run-to-run deterministic on MUSA.
  • The 64-token hybrid-attention fast path is mutually exclusive with an explicit --block-size; an explicit value warns and disables the fast path.
  • FLASH_ATTN has no per-request causal: MUSA has no FA4 dynamic_causal and MATE FA3 types causal as a bool, so a tensor causal raises (reject_per_sequence_causal). Diffusion models (model_config.is_diffusion) need per-request causal across encoder and denoise requests, so they are pinned to TRITON_ATTN; an explicit --attention-backend FLASH_ATTN is overridden with a warning, while any other explicit backend is left as requested. (#254)
  • MLA: MUSA MLA prefill is hard-wired to MATE FlashAttention, and output_scale is still unimplemented. GLM DSA MATE sparse prefill only covers the 64-head / 576-width / index_topk=2048 shape.
  • MinerU dependency: mineru-vl-utils is required on both the engine and the client to run its logits processor; a plain chat completion does not trigger that pipeline.
  • Model Runner: MUSA and DeepSeek-V4 default to Model Runner V1 (unchanged in this release); DSpark uses the V2 override, and any other use of V2 must be opted in explicitly and re-validated for graph capture.
  • Cookbook parameters: target an approximately 4K-input / 1K-output bounded-concurrency workload. Re-tune max-num-batched-tokens, max-num-seqs and graph-capture settings for other request shapes.
  • GLM-5.3 no-thinking behavior relies on the cookbook template workaround; GLM-5.2 uses its checkpoint-provided template.
  • The DeepSeek-V4-Flash-0731 1P1D profile requires RoCE, the Mooncake MUSA wheel, an SSD offload directory, about 400 GiB of free host DRAM (128K profile) and ulimit -l unlimited.
  • FP8, MTP and CUDA Graph paths are sensitive to the model revision, dependency versions and input shapes. Validate a real request and semantic output before production deployment.