Repository navigation
vLLM-MUSA v0.28.0 Release Notes
https://docs.mthreads.com/vllm-musa/vllm-musa-doc-online/S5000/releasenote/
中文版
概览
vLLM-MUSA v0.28.0 对齐上游 vLLM 0.28.0(pin 2cf0a69),是 v0.24.0 之后的下一个发布版本;相对 v0.24.0 分支(2733b6f),v0.28.0-dev 包含 70 个提交、9 位贡献者。本版本把引擎基线推进到 v0.28,新增 NVIDIA-Nemotron-3.5-Lightning-30B-A3B、GLM-5.3、DeepSeek-V4-Flash-0731 以及四款 OCR 模型的 S5000 服务支持,补齐 AWQ/GPTQ W4A16 量化、Mooncake 1P1D 分离式服务与混合 KV cache 快速路径,并把软件栈升级到 PyTorch 2.11.0.post2 / MATE 0.2.7 / torchada 0.1.95。
最终发布镜像
registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0
主要更新
上游基线
- 同步上游 vLLM 0.28.0,适配新的 attention、fused-MoE、sampler、MLA、GDN 与 Model Runner 接口;
vllm-musa包版本同步更新为0.1.28。 - Model Runner 选择:MUSA 默认仍使用 Model Runner V1(DeepSeek-V4 的
FULL_DECODE_ONLY路径更快),Model Runner V2 保留为显式 opt-in,DSpark 场景另有 V2 override。
新模型支持
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B(hybrid Mamba-2 / attention MoE):提供 TP1 + MTP6 的单请求低延迟配置,以及 TP2 + MTP4、最多 98 并发的吞吐配置,配套 mamba2 SSD prefill tile。 (#234, #238, #258)
- GLM-5.3 FP8 / GLM-5.2:恢复 GLM MLA 路径,支持 TP8 + PP2 + EP 的 16 卡部署与
FULL_DECODE_ONLYgraph 捕获;PP > 1 时 MCCL 改为 eager 初始化;GLM DSA sparse-MLA 的 MATE prefill 仅在 64 head / 576 width /index_topk=2048形态下启用,其余形态回落到通用 TileLang 稀疏路径。 (#236) - DeepSeek-V4-Flash-0731:启用 DSpark4 推测解码与 learned indexer,新增单机与 1P1D 两种配置。 (#239, #269)
- OCR 模型族:DeepSeek-OCR、MinerU2.5-Pro-2605-1.2B、OvisOCR2、PaddleOCR-VL-1.6 均提供 1× S5000 / TP1 的 BF16 配置;视觉 rotary 快路径统一由 optimization contract 门控,DeepSeek-OCR 的视觉 attention 默认走 Triton,并修正其 MUSA 精度问题。 (#212, #266, #267, #268, #315)
- Qwen:新增 Qwen3.5-122B-A10B(TP8 + MTP3)、Qwen3-Next-80B-A3B-Instruct(TP8,piecewise graph)、Qwen3.8-27B(TP2 + MTP3)与 Qwen3.5 小模型(0.8B/2B/4B/9B,MTP1)配置;多模态侧加入 Qwen3-VL MM encoder 的 FlashAttention 路由与可选的视觉块 graph capture。
- 模型专用镜像:随本发布线提供
qwen38-flash-next、glm53-flash、hy4-preview(vllm-openai)与flux3(vllm-omni)等 tag。这些 tag 独立于本分支维护,对应模型未在本分支内验证,使用时请自行核对镜像与模型 revision。
注意力与 KV cache
- attention backend 能力声明:各 backend 必须重新声明上游的
supports_*能力,新增能力一致性测试;MUSAFlashAttentionBackend补齐supports_sliding_window,使 interleaved sliding/full 模型可跑在 FLASH_ATTN 上。 (#254) - diffusion 固定
TRITON_ATTN:diffusion 模型的 encoder/denoise 请求需要逐请求 causal 掩码,而 MATE FA3 的causal是 bool、MUSA 又没有 FA4 的dynamic_causal,所以 FLASH_ATTN 遇到 tensor 形式的causal会直接报错;diffusion 模型即使显式指定--attention-backend FLASH_ATTN也会被覆盖并告警。 (#254) - 混合 attention page-size 快路径:默认固定 64-token block 并输出日志,align 模式的 Mamba block table 由单个 Triton kernel 生成;显式
--block-size不再被静默覆盖,而是给出告警并关闭该快路径。 (#285) - Mamba / 混合 KV:修复 prefix-cache pool 归属、淘汰与 headroom gating,Mooncake 场景使用专用 pool;另补 GDN 全量 decode metadata 复用与 uniform-decode 下跳过 SSM slot-mapping(patch 0101、0103)。 (#221, #226, #235, #247)
- MLA:prefill 由
MUSAMLAPrefillBackend接管(不再走MLACommonImpl的报错分支),chunked-context prefill 迁移到 vLLM v0.28 的 per-request chunk layout;MUSA 上关闭_pad_v,output_scale仍未实现。 (#260) - DeepSeek-V4:新增原生 512-d sparse compressor(含
save_partial合并写入)与 FLASHMLA Q-head pad 融合;稀疏 indexer top-k 使用独立 MUSA op;长 prefill indexer logits 以 512 MiB 预算和 512 行 tile 上限约束;learned indexer 保留在 graph decode;aux-stream overlap 仅在 FULL graph 中启用。 (#245, #253, #283)
量化与 MoE
- AWQ/GPTQ W4A16:在 MUSA 上改用 Triton W4A16 kernel(沿 AMD/ROCm 路线,不引入 Marlin);M > 64 时约束到
BLOCK_M=128 / BLOCK_N=64 / BLOCK_K=32(MUSA 上BLOCK_N=128、BLOCK_K>=64会返回 NaN 且跨次运行不一致),M ≤ 64 与上游逐字节一致。 (#277) - W4A16 decode 优化:split-K(2–8 段、fp32 累加、每段不跨量化组)、
torch.library.custom_op包装(避免 Dynamo 在 trace 时冻结按 M 选 tile 的分支)、gate_up改用 16×64 tile(BLOCK_K=64)。 (patch 0185–0188) - TileLang grouped top-k router:新增 grouped softmax top-k 路由 kernel 的 TileLang 实现(最多 512 experts / 128 groups / top-k 32),带独立 kernel 测试。 (#192)
- Router gate:新增
router_gate_fp32,2-D grid FP32 GEMM 取代 grid=(1,1,1) 的 vendor SGEMM,保持 FMA 累加顺序(bitwise 一致),并提供VLLM_MUSA_ROUTER_GATE_FP32=0关闭开关。 (#241) - tiny-M MoE:M ≤ 32 的默认配置从 4 个 pipeline stage 改为 1 个。 (#242)
- DeepGEMM:适配 MATE 0.2.6+ 的 grouped wrapper。 (#207)
推测解码
- DSpark 与 MTP 组合:新增 DSpark(DeepSeek-V4-Flash-0731)与 Nemotron MTP6,并覆盖 Qwen3.5 MTP1、Qwen3.6 MTP2/MTP3、Qwen3.8 MTP3。
- Model Runner V2 sampler:rejection sampler kernel 适配 MUSA Triton 的 scalar predicate 与 Gumbel helper 约定,不改变上游接受/重采样算法。
- async MTP:修复 metadata 归属与 state 就绪事件。
性能
以下数字取自相关提交的测量记录(本发布流程未重新跑测,也不是 v0.24.0 → v0.28.0 的整体基准;除注明外为 S5000):
| 变更 | 测量结果 | 条件 |
|---|---|---|
| W4A16 decode (#277, patch 0185/0187/0188) | M=1 时 o_proj/down_proj 只暴露 24–64 个 program、带宽 7–13 GB/s(峰值 1570 GB/s),split-K 拆 2–8 段以抬高 program 数;gate_up 的 decode tile 由 32×32 改为 16×64(BLOCK_K=64),带宽 100 → 240 GB/s |
S5000 kernel 级测量,数字取自 patch 0185/0188 的提交说明 |
| Router gate FP32 并行 GEMM (#241) | kernel 1.77×/1.92×/2.04×(M=1/7/115),M=512 为 1.02×,bitwise 一致;E2E TPOT 8.1576 → 7.49 ms/token(-8.2%),总时长 9.5808 → 8.9134 s(-7.0%),TTFT 1.4294 → 1.4321 s(持平),输出 SHA 一致 | Nemotron-3.5 MTP6,TP1,batch 1,4096-in/1000-out,1 warmup + 3 次测量 |
| tiny-M MoE 单 stage (#242, patch 0168) | 11.635 → 9.563 ms/token(-17.80%),输出逐字节一致(9 个请求同一 SHA),TTFT 基本持平(1648 vs 1678 ms) | S5000,4096-in/128-out,单请求,Nemotron-3.5 MTP6 decode 形状 |
| mamba2 SSD prefill tile 8 个 warp (#258) | 单次调用 1476 → 756 µs,TTFT 1028.03 → 898.55 ms(-12.59%) | chunk 128 / hdim 64 / dstate 128 / groups ratio 8 |
此外还有一批未附具体倍数的性能改动:DeepSeek-V4 长上下文高并发 prefill、1P1D 下的 prefill/decode 调优(#269, #275)、交错 MRoPE 改为 strided copy 重建(#246)、Model Runner V2 按 in-flight step 轮转 output copy stream(#263)、capability 检查不再重复初始化 MTML(#278)。
分离式服务(PD / Mooncake)
- Mooncake wheel:升级到
0.3.13。 - K/V-first cache layout:Mooncake 接受 MUSA FlashAttention 的 K/V-first 布局(独立 dense K/V 区域,或 padded-page-aware 的 blocks-first 混合视图),不改变 kernel 侧 cache 格式。 (#215)
- KV load 失败上报:失败会回报给上层,不再静默丢失。 (#276)
- 示例:新增
example/disaggregated_serving/,含 Mooncake store offload 与 1P1D(MooncakeConnector+MooncakeStoreConnector)可直接运行的脚本与前置条件说明。 (#264)
构建、补丁系列与镜像
- 补丁规模:构建期 patch series 由 122 个增至 194 个,并新增
patches/series/upstream/(3 个 upstream cherry-pick,上游 pin 升级后自动剔除已合入的补丁)。 - 补丁系列:改为
git am -3可回放,保留indexblob 行、连续编号,并由tools/musa_sync.py check-series --replay/--round-trip校验 mailbox 形状、真实回放与 regen 字节不动点;module-drift tripwire 覆盖到 MLA common 模块。 (#250) - patch 0138:重新生成(Mamba prefix-cache pool 归属)。 (#289)
- Docker 构建:vendored vLLM 依赖安装加
--constraint requirements/common.txt;安装后审计覆盖完整 MUSA cohort(mate、mate-mubin、flash_attn_3、flash_mla、deep-gemm、flashinfer-python、sageattention、tilelang_musa、apache-tvm-ffi、torch-c-dlpack-ext)。 - 镜像元数据:新增 OCI label
com.mthreads.vllm-musa.mooncake-version;tag flavor 更新为..._pytorch_release_2.11.0.post2_musa5.2.0。 - 启动路径:custom op 在 AOT cache load 之前预加载;CUDA graph capture 前后正确处理 expandable segments,并保留其余 allocator 设置;MTML 生命周期与 vLLM v0.28 对齐。 (#284, #291, #298)
文档与示例
- Cookbook:扩展到 Qwen / DeepSeek / GLM / Nemotron / OCR 五个族,每行统一标注精度、硬件、推测解码与 recipe。
- 示例:新增
example/generate/(Qwen3.5-35B-A3B-FP8 离线与在线示例);docs/example/由example/disaggregated_serving/取代(README 重写,并加入 Mooncake store 与 1P1D 的可运行脚本)。 - 安装文档:补充双 index 安装方式;
mineru-vl-utils纳入运行时依赖。
Cookbook 配置
以下配置来自 v0.28.0 serving cookbook,面向 S5000、约 4K 输入 / 1K 输出的有界并发场景。请把同一行的精度、TP 和推测解码设置作为一个整体复制;更换模型、并发或上下文长度后需要重新校准。
默认镜像:registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0
Qwen
| 模型 | 精度 | 硬件 / 并行 | 推测解码 |
|---|---|---|---|
| Qwen3-8B | FP8 | 1× S5000 / TP1 | Off |
| Qwen3.5-0.8B | BF16 | 1× S5000 / TP1 | MTP1 |
| Qwen3.5-2B | BF16 | 1× S5000 / TP1 | MTP1 |
| Qwen3.5-4B | BF16 | 1× S5000 / TP1 | MTP1 |
| Qwen3.5-9B | BF16 | 2× S5000 / TP2 | MTP1 |
| Qwen3.5-27B | BF16 / FP8 | 2× S5000 / TP2 | MTP1 |
| Qwen3.5-35B-A3B | BF16 / FP8 | 4× S5000 / TP4 | MTP3 |
| Qwen3.5-122B-A10B | BF16 / FP8 | 8× S5000 / TP8 | MTP3 |
| Qwen3.5-397B-A17B | FP8 | 8× S5000 / TP8 | Off |
| Qwen3.6-27B | BF16 / FP8 | 2× S5000 / TP2 | MTP3 |
| Qwen3.6-35B-A3B | BF16 / FP8 | 4× S5000 / TP4 | MTP2 |
| Qwen3.8-27B | BF16 / FP8 | 2× S5000 / TP2 | MTP3 |
| Qwen3-Next-80B-A3B-Instruct | BF16 | 8× S5000 / TP8 | Off |
| Qwen3-VL-8B-Instruct | BF16 / FP8 | 1× S5000 / TP1 | Off |
| Qwen3-VL-30B-A3B-Instruct | BF16 / FP8 | 4× S5000 / TP4 | Off |
| Qwen3-VL-32B-Instruct | BF16 / FP8 | 4× S5000 / TP4 | Off |
DeepSeek / GLM / Nemotron / OCR
| 模型 | 精度 | 硬件 / 并行 | 推测解码 |
|---|---|---|---|
| DeepSeek-V4-Flash | FP8 KV cache | 8× S5000 / TP8 | Fixed MTP4(MTP-off 可选) |
| DeepSeek-V4-Flash-0731 | FP8 | 8× S5000 / TP8(1P1D 时为每节点 8 卡) | DSpark4(可选 1P1D + Mooncake store) |
| GLM-5.3 / GLM-5.2 | FP8 | 16× S5000 / TP8 + PP2(EP 开启,FULL_DECODE_ONLY) |
— |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | BF16 | 1× S5000 / TP1 | MTP6 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | BF16 | 2× S5000 / TP2 | MTP4(最多 98 并发) |
| DeepSeek-OCR | BF16 | 1× S5000 / TP1 | — |
| MinerU2.5-Pro-2605-1.2B | BF16 | 1× S5000 / TP1 | — |
| OvisOCR2 | BF16 | 1× S5000 / TP1 | — |
| PaddleOCR-VL-1.6 | BF16 | 1× S5000 / TP1 | — |
表中
Off表示 cookbook 明确关闭推测解码,—表示该 recipe 未涉及。Nemotron-3.5-Lightning 的两个配置是延迟与吞吐的取舍:TP1(--max-num-seqs 1,MTP6)面向单请求的逐 token 延迟,需要并发请改用 TP2(MTP4,最多 98 并发);两个配置的启动参数应整体复制。
软件栈
| 组件 | v0.24.0 | v0.28.0 |
|---|---|---|
| vLLM(上游 pin) | 0.24.0 (ee0da84) |
0.28.0 (2cf0a69) |
vllm-musa 包 |
0.1.24 | 0.1.28 |
| Python | 3.10 | 3.10 |
| PyTorch / torch_musa | 2.11.0.post1+musa5.2.0 | 2.11.0.post2+musa5.2.0 |
| MATE | 0.2.4 | 0.2.7 |
flash_attn_3 / flash_mla / deep-gemm / sageattention |
0.2.4 | 0.2.7+musa |
mate-mubin |
— | 0.2.7 |
flashinfer-python |
— | 0.2.7+musa |
| torchada | 0.1.83 | 0.1.95 |
deep_ep |
1.1.0+musa5.2.0torch2.11.0.post1 | 1.1.0+musa5.2.0torch2.11.0.post2 |
tilelang_musa |
0.1.8+musa.3 | 0.1.12+musa.2 |
apache-tvm-ffi |
0.1.9.post3+musa.1 | 0.1.11.post1+musa.1 |
| Triton | 3.2.0 | 3.2.0 |
| Transformers | 5.5.3 | 5.15.0 |
mthreads-ml-py |
>=2.4.2 | >=2.4.3 |
| Mooncake MUSA wheel | 0.3.12.post1 | 0.3.13 |
Bug 修复
- 修复 AR+RMSNorm 重写插入节点缺失 fake-tensor meta 的问题。 (#261)
- 修复 Mooncake 未上报失败 KV load 的问题。 (#276)
- 修复 Qwen3 fused op 参数传递与嵌套 multimodal config 缺
architectures的兼容问题。 (#232, #237) - 修复无 GPU 构建环境下的 TileLang import 问题,并收窄其 workaround 范围。 (#214, #219)
- 修复 MinerU 视觉语言运行时依赖缺失(
mineru-vl-utils)。 (#313)
升级说明
推荐升级方式:直接更换镜像,不要手动升级依赖。
v0.28.0 是一次跨越式升级:torch/torch_musa 到 2.11.0.post2、MATE 系列包到 0.2.7、torchada 到 0.1.95、tilelang_musa 到 0.1.12、transformers 到 5.15.0。这些包之间有版本耦合(MATE 与 flash_attn_3/flash_mla/deep-gemm/flashinfer-python 同批次发布,tilelang_musa 绑定 apache-tvm-ffi),镜像构建时会对整个 cohort 做一次版本审计,因此换镜像是唯一省事且不会漏包的路径:
docker pull registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0- 不建议在旧镜像里
pip install -U vllm-musa:这样只会升插件包,torch_musa、MATE、torchada、tilelang_musa仍是旧版本,容易装出官方未验证的混搭组合。 - 必须从源码构建时,请按
docs/installation.md的双 index 顺序整体安装(先musa_private.txt+--no-deps,再build.txt/common.txt,最后pip install . --no-build-isolation),不要逐个升级包。 - 版本标识:上游 vLLM 0.28.0(pin
2cf0a69),vllm-musa插件包 metadata 版本0.1.28。 - 设备可见性:由部署环境显式设置
MUSA_VISIBLE_DEVICES,镜像不预置固定设备列表。 - 运行时判断:MUSA 环境不要用
torch.cuda.is_available(),应使用项目提供的 MUSA platform helper。
已知限制
- W4A16 tile 约束:M > 64 时 Triton W4A16 固定
BLOCK_M=128 / BLOCK_N=64 / BLOCK_K=32;BLOCK_N=128或BLOCK_K>=64的 tile 在 MUSA 上会返回 NaN,且跨次运行不一致。 - 混合 attention 的 64-token 快路径与显式
--block-size互斥,显式指定时会告警并关闭快路径。 - FLASH_ATTN 不支持逐请求 causal:MUSA 没有 FA4 的
dynamic_causal,MATE FA3 的causal是 bool,因此遇到 tensorcausal会直接报错(reject_per_sequence_causal)。diffusion 模型(model_config.is_diffusion)的 encoder/denoise 请求需要逐请求 causal,因此被固定到TRITON_ATTN;显式指定--attention-backend FLASH_ATTN会被覆盖并告警,其他显式 backend 保持原样。 (#254) - MLA:MUSA 的 MLA prefill 固定走 MATE FlashAttention;
output_scale仍未实现。GLM DSA 的 MATE sparse prefill 仅覆盖 64 head / 576 width /index_topk=2048形态。 - MinerU 依赖:engine 与 client 两侧都需要
mineru-vl-utils才能运行其 logits processor;普通 chat completion 不会触发该 pipeline。 - Model Runner:MUSA 与 DeepSeek-V4 默认使用 Model Runner V1(本版本未改变);DSpark 场景使用 V2 override,其他情况需要 V2 请显式指定并按模型重新验证 graph 捕获。
- Cookbook 参数:面向约 4K 输入 / 1K 输出的有界并发场景;其他请求形状需要重新调整
max-num-batched-tokens、max-num-seqs和 graph capture 配置。 - GLM-5.3 的 no-thinking 行为依赖 cookbook 中的模板 workaround;GLM-5.2 使用 checkpoint 自带模板。
- DeepSeek-V4-Flash-0731 的 1P1D 配置需要 RoCE、Mooncake MUSA wheel、SSD offload 目录、约 400 GiB 空闲 host DRAM(128K 配置)与
ulimit -l unlimited。 - FP8、MTP 和 CUDA Graph 路径对模型 revision、依赖版本和输入 shape 敏感,生产部署前应完成真实请求与语义检查。
English Version
Overview
vLLM-MUSA v0.28.0 aligns with upstream vLLM 0.28.0 (pinned at 2cf0a69) and is the next release after v0.24.0; against the v0.24.0 branch (2733b6f), v0.28.0-dev carries 70 commits from 9 contributors. It moves the engine baseline to v0.28, adds S5000 serving support for NVIDIA-Nemotron-3.5-Lightning-30B-A3B, GLM-5.3, DeepSeek-V4-Flash-0731 and four OCR models, enables AWQ/GPTQ W4A16 quantization, Mooncake 1P1D disaggregated serving and a mixed-pool KV cache fast path, and upgrades the stack to PyTorch 2.11.0.post2 / MATE 0.2.7 / torchada 0.1.95.
Final Release Image
registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0
Highlights
Upstream alignment
- Synchronized with upstream vLLM 0.28.0, adapting the attention, fused-MoE, sampler, MLA, GDN and Model Runner interfaces; the
vllm-musapackage version is now0.1.28. - Model Runner selection: MUSA still defaults to Model Runner V1 (the DeepSeek-V4
FULL_DECODE_ONLYpath is faster there), with Model Runner V2 kept as an explicit opt-in and a separate V2 override for DSpark.
New model support
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B (hybrid Mamba-2 / attention MoE): a TP1 + MTP6 profile tuned for single-request decode latency and a TP2 + MTP4 profile admitting up to 98 concurrent sequences, backed by mamba2 SSD prefill tiles. (#234, #238, #258)
- GLM-5.3 FP8 / GLM-5.2: restores the GLM MLA path and supports 16-GPU TP8 + PP2 + EP deployments with
FULL_DECODE_ONLYgraph capture; MCCL init is eager when PP > 1; GLM DSA sparse-MLA MATE prefill is enabled only for the 64-head / 576-width /index_topk=2048shape, with everything else on the generic TileLang sparse route. (#236) - DeepSeek-V4-Flash-0731: enables DSpark4 speculative decoding with the learned indexer, in single-node and 1P1D shapes. (#239, #269)
- OCR family: DeepSeek-OCR, MinerU2.5-Pro-2605-1.2B, OvisOCR2 and PaddleOCR-VL-1.6 each ship a 1× S5000 / TP1 BF16 profile; vision rotary fast paths are gated by the optimization contract, DeepSeek-OCR vision attention defaults to Triton, and its MUSA precision issue is fixed. (#212, #266, #267, #268, #315)
- Qwen: new Qwen3.5-122B-A10B (TP8 + MTP3), Qwen3-Next-80B-A3B-Instruct (TP8, piecewise graphs), Qwen3.8-27B (TP2 + MTP3) and small Qwen3.5 models (0.8B/2B/4B/9B, MTP1) profiles; the multimodal side gains FlashAttention routing for Qwen3-VL MM encoders and optional vision-block graph capture.
- Model-specific images: this release line also publishes tags
qwen38-flash-next,glm53-flash,hy4-preview(vllm-openai) andflux3(vllm-omni). These tags are maintained independently of this branch and their models are not validated in it; verify both the image and the model revision yourself.
Attention and KV cache
- Attention backend capabilities: every backend must re-declare the upstream
supports_*capabilities, backed by a new capability-parity test;MUSAFlashAttentionBackendgainedsupports_sliding_window, so interleaved sliding/full models can run on FLASH_ATTN. (#254) - Diffusion is pinned to
TRITON_ATTN: diffusion encoder/denoise requests need per-request causal masks, but MATE FA3 typescausalas a bool and MUSA has no FA4dynamic_causal, so FLASH_ATTN raises on a tensorcausal; an explicit--attention-backend FLASH_ATTNis overridden with a warning. (#254) - Hybrid attention page-size fast path: the 64-token block is pinned by default and logged, and align-mode Mamba block tables are built by a single Triton kernel; an explicit
--block-sizeis no longer silently overridden — it warns and disables the fast path. (#285) - Mamba / hybrid KV: fixed prefix-cache pool ownership, eviction and headroom gating, with a dedicated pool for Mooncake, plus common GDN full-decode metadata reuse and SSM slot-mapping elision in uniform decode (patches 0101, 0103). (#221, #226, #235, #247)
- MLA: prefill is now owned by
MUSAMLAPrefillBackend(no longer theMLACommonImplerror branch), and chunked-context prefill migrated to the vLLM v0.28 per-request chunk layout;_pad_vis disabled on MUSA andoutput_scaleremains unimplemented. (#260) - DeepSeek-V4: native 512-d sparse compressor (folding in the
save_partialstate write) and FLASHMLA Q-head pad fusion; sparse indexer top-k runs on a dedicated MUSA op; long-prefill indexer logits are bounded by a 512 MiB budget and a 512-row tile cap; the learned indexer is retained on graph decode; aux-stream overlap is confined to FULL graphs. (#245, #253, #283)
Quantization and MoE
- AWQ/GPTQ W4A16 on MUSA now uses the Triton W4A16 kernel (the AMD/ROCm route, no Marlin port); for M > 64 the tile is constrained to
BLOCK_M=128 / BLOCK_N=64 / BLOCK_K=32(BLOCK_N=128andBLOCK_K>=64return NaN and are not run-to-run deterministic on MUSA), while M ≤ 64 stays byte-identical to upstream. (#277) - W4A16 decode work: split-K (2–8 pieces, fp32 reduction, no piece crossing a quantization group), a
torch.library.custom_opwrapper so Dynamo cannot freeze the per-M tile branch at trace time, and a 16×64 tile withBLOCK_K=64forgate_up. (patches 0185–0188) - TileLang grouped top-k router: a TileLang implementation of the grouped softmax top-k route (up to 512 experts / 128 groups / top-k 32), with its own kernel tests. (#192)
- Router gate: the new
router_gate_fp322-D-grid FP32 GEMM replaces the grid=(1,1,1) vendor SGEMM while preserving FMA accumulation order (bitwise identical), with aVLLM_MUSA_ROUTER_GATE_FP32=0kill switch. (#241) - tiny-M MoE: the default for M ≤ 32 moves from 4 pipeline stages to 1. (#242)
- DeepGEMM: adapted the grouped wrappers to MATE 0.2.6+. (#207)
Speculative decoding
- DSpark and MTP coverage: added DSpark (DeepSeek-V4-Flash-0731) and Nemotron MTP6, and covered Qwen3.5 MTP1, Qwen3.6 MTP2/MTP3 and Qwen3.8 MTP3.
- Model Runner V2 sampler: rejection-sampler kernels were adapted to the MUSA Triton scalar-predicate and Gumbel-helper contracts without changing upstream acceptance or resampling.
- async MTP: fixed metadata ownership and state-readiness events.
Performance
Numbers below are taken from the cited commits (not rerun for this release, and not a v0.24.0 → v0.28.0 benchmark; S5000 unless noted):
| Change | Measured | Conditions |
|---|---|---|
| W4A16 decode (#277, patches 0185/0187/0188) | At M=1, o_proj/down_proj expose only 24–64 programs and stream at 7–13 GB/s against a 1570 GB/s peak, so split-K breaks K into 2–8 pieces; the gate_up decode tile moves from 32×32 to 16×64 with BLOCK_K=64, lifting bandwidth from 100 to 240 GB/s |
S5000 kernel-level measurements quoted from the patch 0185/0188 commit messages |
| Parallel FP32 router-gate GEMM (#241) | Kernel 1.77×/1.92×/2.04× (M=1/7/115), 1.02× at M=512, bitwise identical; end-to-end TPOT 8.1576 → 7.49 ms/token (-8.2%), total time 9.5808 → 8.9134 s (-7.0%), TTFT 1.4294 → 1.4321 s (unchanged), identical output SHA | Nemotron-3.5 MTP6, TP1, batch 1, 4096-in/1000-out, 1 warmup + 3 measured runs |
| One pipeline stage for tiny-M MoE (#242, patch 0168) | 11.635 → 9.563 ms/token (-17.80%), byte-identical output (same SHA across 9 requests), TTFT unchanged (1648 vs 1678 ms) | S5000, 4096-in/128-out, one request, Nemotron-3.5 MTP6 decode shape |
| mamba2 SSD prefill tile at eight warps (#258) | 1476 → 756 µs per call, TTFT 1028.03 → 898.55 ms (-12.59%) | chunk 128 / hdim 64 / dstate 128 / groups ratio 8 |
A further set of performance changes ships without published multipliers: DeepSeek-V4 long-context high-concurrency prefill and 1P1D prefill/decode tuning (#269, #275), interleaved MRoPE rebuilt with a strided copy (#246), Model Runner V2 rotating its output copy stream per in-flight step (#263), and capability checks that no longer reinitialize MTML (#278).
Disaggregated serving (PD / Mooncake)
- Mooncake wheel: upgraded to
0.3.13. - K/V-first cache layout: Mooncake accepts MUSA FlashAttention's K/V-first cache layout (separate dense K/V regions, or a padded-page-aware blocks-first hybrid view) without changing the kernel-facing cache format. (#215)
- KV load failures: now reported upward instead of being dropped silently. (#276)
- Examples: new
example/disaggregated_serving/with runnable Mooncake store-offload and 1P1D (MooncakeConnector+MooncakeStoreConnector) scripts with their prerequisites. (#264)
Build, patch series and image
- Patch volume: the build-time series grows from 122 to 194 patches, plus a new
patches/series/upstream/(3 upstream cherry-picks, auto-dropped once the upstream pin already contains them). - Patch series: now
git am -3replayable —indexblob lines are preserved, numbering stays contiguous, andtools/musa_sync.py check-series --replay/--round-tripvalidates mailbox shape, a real replay and the regen byte fixed point; module-drift tripwires now cover the MLA common module. (#250) - Patch 0138: regenerated (Mamba prefix-cache pool ownership). (#289)
- Docker build: vendored vLLM dependencies install with
--constraint requirements/common.txt, and the post-install audit covers the full MUSA cohort (mate,mate-mubin,flash_attn_3,flash_mla,deep-gemm,flashinfer-python,sageattention,tilelang_musa,apache-tvm-ffi,torch-c-dlpack-ext). - Image metadata: new OCI label
com.mthreads.vllm-musa.mooncake-version; the tag flavor is now..._pytorch_release_2.11.0.post2_musa5.2.0. - Startup: custom ops are preloaded before the AOT cache load; expandable segments are handled around graph capture while the rest of the allocator configuration is preserved; MTML lifecycle handling is aligned with vLLM v0.28. (#284, #291, #298)
Documentation and examples
- Cookbook: now spans five families — Qwen, DeepSeek, GLM, Nemotron and OCR — with precision, hardware, speculative decoding and a recipe for every row.
- Examples: added
example/generate/(Qwen3.5-35B-A3B-FP8 offline and online examples);docs/example/is replaced byexample/disaggregated_serving/(rewritten README plus runnable Mooncake store and 1P1D scripts). - Installation docs: cover the dual-index install;
mineru-vl-utilsis now a runtime dependency.
Documented Cookbook Profiles
The profiles below are from the v0.28.0 serving cookbook and target S5000 hardware, approximately 4K input / 1K output, and bounded concurrency. Copy the precision, TP and speculative-decoding settings of a row as a unit; changing the model, concurrency or context length requires recalibration.
Default image: registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0
Qwen
| Model | Precision | Hardware / parallelism | Speculative decoding |
|---|---|---|---|
| Qwen3-8B | FP8 | 1× S5000 / TP1 | Off |
| Qwen3.5-0.8B | BF16 | 1× S5000 / TP1 | MTP1 |
| Qwen3.5-2B | BF16 | 1× S5000 / TP1 | MTP1 |
| Qwen3.5-4B | BF16 | 1× S5000 / TP1 | MTP1 |
| Qwen3.5-9B | BF16 | 2× S5000 / TP2 | MTP1 |
| Qwen3.5-27B | BF16 / FP8 | 2× S5000 / TP2 | MTP1 |
| Qwen3.5-35B-A3B | BF16 / FP8 | 4× S5000 / TP4 | MTP3 |
| Qwen3.5-122B-A10B | BF16 / FP8 | 8× S5000 / TP8 | MTP3 |
| Qwen3.5-397B-A17B | FP8 | 8× S5000 / TP8 | Off |
| Qwen3.6-27B | BF16 / FP8 | 2× S5000 / TP2 | MTP3 |
| Qwen3.6-35B-A3B | BF16 / FP8 | 4× S5000 / TP4 | MTP2 |
| Qwen3.8-27B | BF16 / FP8 | 2× S5000 / TP2 | MTP3 |
| Qwen3-Next-80B-A3B-Instruct | BF16 | 8× S5000 / TP8 | Off |
| Qwen3-VL-8B-Instruct | BF16 / FP8 | 1× S5000 / TP1 | Off |
| Qwen3-VL-30B-A3B-Instruct | BF16 / FP8 | 4× S5000 / TP4 | Off |
| Qwen3-VL-32B-Instruct | BF16 / FP8 | 4× S5000 / TP4 | Off |
DeepSeek / GLM / Nemotron / OCR
| Model | Precision | Hardware / parallelism | Speculative decoding |
|---|---|---|---|
| DeepSeek-V4-Flash | FP8 KV cache | 8× S5000 / TP8 | Fixed MTP4 (MTP-off available) |
| DeepSeek-V4-Flash-0731 | FP8 | 8× S5000 / TP8 (8 GPUs per node in 1P1D) | DSpark4 (optional 1P1D + Mooncake store) |
| GLM-5.3 / GLM-5.2 | FP8 | 16× S5000 / TP8 + PP2 (EP enabled, FULL_DECODE_ONLY) |
— |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | BF16 | 1× S5000 / TP1 | MTP6 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | BF16 | 2× S5000 / TP2 | MTP4 (up to 98 concurrent sequences) |
| DeepSeek-OCR | BF16 | 1× S5000 / TP1 | — |
| MinerU2.5-Pro-2605-1.2B | BF16 | 1× S5000 / TP1 | — |
| OvisOCR2 | BF16 | 1× S5000 / TP1 | — |
| PaddleOCR-VL-1.6 | BF16 | 1× S5000 / TP1 | — |
In the tables,
Offmeans the cookbook explicitly disables speculative decoding, while—means the recipe does not cover it. The two Nemotron-3.5-Lightning profiles trade throughput for latency: TP1 (--max-num-seqs 1, MTP6) targets per-token latency for a single request, so concurrent traffic should use TP2 (MTP4, up to 98 sequences). Copy each profile's launch arguments as a unit.
Software Stack
| Component | v0.24.0 | v0.28.0 |
|---|---|---|
| vLLM (upstream pin) | 0.24.0 (ee0da84) |
0.28.0 (2cf0a69) |
vllm-musa package |
0.1.24 | 0.1.28 |
| Python | 3.10 | 3.10 |
| PyTorch / torch_musa | 2.11.0.post1+musa5.2.0 | 2.11.0.post2+musa5.2.0 |
| MATE | 0.2.4 | 0.2.7 |
flash_attn_3 / flash_mla / deep-gemm / sageattention |
0.2.4 | 0.2.7+musa |
mate-mubin |
— | 0.2.7 |
flashinfer-python |
— | 0.2.7+musa |
| torchada | 0.1.83 | 0.1.95 |
deep_ep |
1.1.0+musa5.2.0torch2.11.0.post1 | 1.1.0+musa5.2.0torch2.11.0.post2 |
tilelang_musa |
0.1.8+musa.3 | 0.1.12+musa.2 |
apache-tvm-ffi |
0.1.9.post3+musa.1 | 0.1.11.post1+musa.1 |
| Triton | 3.2.0 | 3.2.0 |
| Transformers | 5.5.3 | 5.15.0 |
mthreads-ml-py |
>=2.4.2 | >=2.4.3 |
| Mooncake MUSA wheel | 0.3.12.post1 | 0.3.13 |
Bug Fixes
- Fixed missing fake-tensor meta on nodes inserted by the AR+RMSNorm rewrite. (#261)
- Fixed Mooncake not reporting failed KV loads. (#276)
- Fixed Qwen3 fused-op argument passing and the compatibility of nested multimodal configs without
architectures. (#232, #237) - Fixed the TileLang import failure on GPU-less build hosts and narrowed the workaround. (#214, #219)
- Fixed the missing MinerU vision-language runtime dependency (
mineru-vl-utils). (#313)
Upgrade Notes
Upgrade by switching the image. Do not upgrade the dependencies by hand.
v0.28.0 is a step change: torch/torch_musa move to 2.11.0.post2, the MATE package family to 0.2.7, torchada to 0.1.95, tilelang_musa to 0.1.12 and transformers to 5.15.0. These packages are coupled (MATE ships in lockstep with flash_attn_3 / flash_mla / deep-gemm / flashinfer-python, and tilelang_musa pins apache-tvm-ffi), and the image build audits the whole cohort. Switching the image is the only path that is both simple and complete:
docker pull registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.28.0- Do not run
pip install -U vllm-musainside an older image: that upgrades the plugin only, leavingtorch_musa, MATE,torchadaandtilelang_musabehind and producing a combination nobody validated. - If you must build from source, follow the dual-index order in
docs/installation.mdas a whole (musa_private.txtwith--no-deps, thenbuild.txt/common.txt, thenpip install . --no-build-isolation) instead of upgrading packages one by one. - Version identifiers: upstream vLLM 0.28.0 (pin
2cf0a69);vllm-musaplugin metadata version0.1.28. - Device visibility: set
MUSA_VISIBLE_DEVICESexplicitly in the deployment environment; the image does not bake in a fixed device list. - Runtime detection: do not use
torch.cuda.is_available()to detect MUSA availability; use the project's MUSA platform helper.
Known Limitations
- W4A16 tile constraint: for M > 64 the Triton W4A16 kernel is fixed to
BLOCK_M=128 / BLOCK_N=64 / BLOCK_K=32; tiles withBLOCK_N=128orBLOCK_K>=64return NaN and are not run-to-run deterministic on MUSA. - The 64-token hybrid-attention fast path is mutually exclusive with an explicit
--block-size; an explicit value warns and disables the fast path. - FLASH_ATTN has no per-request causal: MUSA has no FA4
dynamic_causaland MATE FA3 typescausalas a bool, so a tensorcausalraises (reject_per_sequence_causal). Diffusion models (model_config.is_diffusion) need per-request causal across encoder and denoise requests, so they are pinned toTRITON_ATTN; an explicit--attention-backend FLASH_ATTNis overridden with a warning, while any other explicit backend is left as requested. (#254) - MLA: MUSA MLA prefill is hard-wired to MATE FlashAttention, and
output_scaleis still unimplemented. GLM DSA MATE sparse prefill only covers the 64-head / 576-width /index_topk=2048shape. - MinerU dependency:
mineru-vl-utilsis required on both the engine and the client to run its logits processor; a plain chat completion does not trigger that pipeline. - Model Runner: MUSA and DeepSeek-V4 default to Model Runner V1 (unchanged in this release); DSpark uses the V2 override, and any other use of V2 must be opted in explicitly and re-validated for graph capture.
- Cookbook parameters: target an approximately 4K-input / 1K-output bounded-concurrency workload. Re-tune
max-num-batched-tokens,max-num-seqsand graph-capture settings for other request shapes. - GLM-5.3 no-thinking behavior relies on the cookbook template workaround; GLM-5.2 uses its checkpoint-provided template.
- The DeepSeek-V4-Flash-0731 1P1D profile requires RoCE, the Mooncake MUSA wheel, an SSD offload directory, about 400 GiB of free host DRAM (128K profile) and
ulimit -l unlimited. - FP8, MTP and CUDA Graph paths are sensitive to the model revision, dependency versions and input shapes. Validate a real request and semantic output before production deployment.