v0.24.0

vLLM-MUSA v0.24.0 Release Notes

https://docs.mthreads.com/vllm-musa/vllm-musa-doc-online/S5000/releasenote/

中文版

概览

vLLM-MUSA v0.24.0 对齐上游 vLLM 0.24.0，持续完善 V1 引擎、MUSA 原生算子、注意力与 KV cache、推测解码，以及 Qwen 和 DeepSeek-V4-Flash-Base 的服务路径。本版本同时更新到 PyTorch 2.11.x MUSA 软件栈，并提供一组经过验证的 S5000 serving cookbook 配置。

最终发布镜像

本版本的最终 Docker 镜像为：

registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.24.0

Cookbook 中的所有启动示例均以该镜像为基准；使用等价的本地安装时，请确保软件栈和依赖版本一致。

主要更新

• 上游版本对齐：同步上游 vLLM 0.24.0，适配新的 attention、fused-MoE、sampler 和 Model Runner 接口。
• V1 与编译运行时：完善 V1 引擎和 Model Runner V2 路径，改进 CUDA Graph、torch.compile cache 及首次编译流程。
• Attention 与 KV cache：支持 MUSA FlashAttention 的 FP8 E4M3 KV cache，更新 attention block size，改进混合 batch prefill、KV split 和 DeepSeek-V4-Flash-Base sparse MLA 路径。
• DeepSeek-V4-Flash-Base 长 prefill：优化长上下文 prefill indexer，支持多卡并行和 native C4 prefill compression，并增强 graph 路径的稳定性。
• MoE 与量化：加入 FP8 per-token-group quantization、Qwen3.5 shared-expert 融合、S5000 FP8 MoE backend 选择，以及小 batch BF16 GEMV 优化。
• GDN / Mamba：为 Qwen3.5 GDN decode 引入 fused recurrent update，并优化 Mamba cache layout。
• 推测解码：支持 MUSA sampled speculative decoding，完善 DeepSeek-V4-Flash-Base MTP、Qwen3.5 MTP 和 Eagle3 的 graph metadata、随机数与验证流程。
• 模型适配：增加 GLM-5.2 FP8 和 Qwen Omni ModelOpt FP8 支持，并继续完善 Qwen3.5、Qwen3.6、DeepSeek-V4-Flash-Base 及相关模型路径。
• 构建与交付：改进 Docker 镜像构建和 OCI provenance labels，支持 Mooncake MUSA wheel，并完善 Python 3.10 和双 pip index 安装文档。

已验证的 Cookbook 配置

以下配置来自 v0.24.0 serving cookbook，面向 S5000、约 4K input / 1K output 和 bounded concurrency 场景。请完整复制同一行中的精度、TP 和推测解码设置；更换模型、并发或上下文长度后需要重新校准。

默认镜像：registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.24.0

模型	精度	硬件 / 并行	推测解码
Qwen3-8B	FP8	1× S5000 / TP1	Off
Qwen3.5-27B	BF16	2× S5000 / TP2	MTP1
Qwen3.5-27B	FP8	2× S5000 / TP2	Off
Qwen3.5-35B-A3B	BF16	4× S5000 / TP4	MTP3
Qwen3.5-35B-A3B	FP8	4× S5000 / TP4	MTP3
Qwen3.6-27B	BF16	2× S5000 / TP2	MTP3
Qwen3.6-27B	FP8	2× S5000 / TP2	MTP3
Qwen3.6-35B-A3B	BF16	4× S5000 / TP4	MTP2
Qwen3.6-35B-A3B	FP8	4× S5000 / TP4	MTP2
DeepSeek-V4-Flash-Base	FP8 KV cache	8× S5000 / TP8	Fixed MTP4（MTP-off 可选）

Bug 修复

• 修复 Qwen3.5 FP8 shared-expert fold 的非幂等行为，避免重复追加 shared route 导致错误结果。
• 修复 DeepSeek-V4-Flash-Base 长 prefill indexer 的内存边界、TP 分片和 no-MTP precision semantics；在 CUDA Graph 路径中正确处理辅助操作的依赖。
• 修复 piecewise graph 中注册 custom all-reduce input 导致的错误路径。
• 修复 DeepSeek-V4-Flash-Base MTP prefill hang，以及 Qwen3.5 启用 MTP 后无法启动的问题。
• 修复 Eagle3 Gumbel RNG bitcast 和 full-graph draft metadata。
• 修复 EP service 退出后的 DeepEP 资源清理。
• 修复 Qwen FP8 graph all-reduce、MUSA PrivateUse1 backend 的 repetition penalty 注册，以及 Triton 3.2 unified-attention decode 兼容性。
• 修复 MATE FA3 API、block-FP8 MoE scale sharding、unquantized MoE 过时参数和无 cuda-python 时 MiniMax RMSNorm 路径的降级行为。
• 修复 GPTQ、fused activation/layernorm-quant custom-op 注册和首次编译相关问题。

其他变更与升级说明

• 软件栈：推荐使用 Python 3.10、PyTorch/torch_musa 2.11.0.post1+musa5.2.0、torchada 0.1.83、MATE/flash_attn_3/flash_mla 0.2.4 和 Triton 3.2.0。通用依赖包括 NumPy 1.26、Transformers 5.5.3 和 mthreads-ml-py>=2.4.2。
• 版本标识：本发布线使用上游 vLLM 0.24.0；vllm-musa 插件包的 Python metadata 版本为 0.1.24。
• 安装：MUSA 镜像基于已有的 MUSA PyTorch 安装 patched vLLM；使用独立 wheel 时请确认依赖完整，避免 pip 将 MUSA PyTorch 替换为 CUDA 版本。
• 设备可见性：由部署环境显式设置 MUSA_VISIBLE_DEVICES，镜像不预置固定的设备列表。
• 运行时判断：MUSA 环境不要使用 torch.cuda.is_available() 判断设备可用性，应使用项目提供的 MUSA platform helper。

已知限制

• Cookbook 参数针对约 4K input / 1K output 的 bounded-concurrency workload；其他请求形状需要重新调整 max-num-batched-tokens、max-num-seqs 和 graph capture 配置。
• DeepSeek-V4-Flash-Base 的 MTP-off 配置在高并发或较低显存环境下可能超出显存；请同时降低 max-num-seqs 和 graph capture 上限后再部署。
• FP8、MTP 和 CUDA Graph 路径对模型 revision、依赖版本和输入 shape 敏感，生产部署前应完成真实请求和语义检查。

English Version

Overview

vLLM-MUSA v0.24.0 aligns with upstream vLLM 0.24.0 and continues to improve the V1 engine, MUSA-native kernels, attention and KV cache, speculative decoding, and production serving paths for Qwen and DeepSeek-V4-Flash-Base. The release line also moves to the PyTorch 2.11.x MUSA stack and provides validated S5000 serving cookbook profiles.

Final Release Image

The final Docker image for this release is:

registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.24.0

All cookbook launch examples are based on this image. When using an equivalent local installation, keep the software stack and dependency versions aligned.

Highlights

• Upstream alignment: synchronized with vLLM 0.24.0 and adapted to the updated attention, fused-MoE, sampler, and Model Runner interfaces.
• V1 and compilation runtime: improved the V1 engine and Model Runner V2 paths, CUDA Graph handling, torch.compile caching, and first-build behavior.
• Attention and KV cache: added FP8 E4M3 KV-cache support for MUSA FlashAttention, updated attention block sizes, and improved mixed-batch prefill, KV splitting, and DeepSeek-V4-Flash-Base sparse-MLA paths.
• DeepSeek-V4-Flash-Base long prefill: optimized the long-context prefill indexer with multi-GPU parallelism and native C4 prefill compression, and improved graph-path stability.
• MoE and quantization: added FP8 per-token-group quantization, Qwen3.5 shared-expert fusion, calibrated FP8 MoE backend selection on S5000, and a small-batch BF16 GEMV path.
• GDN / Mamba: added a fused recurrent update for Qwen3.5 GDN decode and improved the Mamba cache layout.
• Speculative decoding: enabled MUSA sampled speculative decoding and improved DeepSeek-V4-Flash-Base MTP, Qwen3.5 MTP, and Eagle3 graph metadata, RNG, and verification paths.
• Model enablement: added GLM-5.2 FP8 and Qwen Omni ModelOpt FP8 support, while continuing to improve Qwen3.5, Qwen3.6, DeepSeek-V4-Flash-Base, and related model paths.
• Build and delivery: improved Docker image builds and OCI provenance labels, added support for the Mooncake MUSA wheel, and documented Python 3.10 and the dual-index installation flow.

Validated Cookbook Profiles

The profiles below are from the v0.24.0 serving cookbook and target S5000 hardware, approximately 4K input / 1K output, and bounded concurrency. Copy the precision, TP, and speculative-decoding settings together; changing the model, concurrency, or context length requires recalibration.

Default image: registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.24.0

Model	Precision	Hardware / parallelism	Speculative decoding
Qwen3-8B	FP8	1× S5000 / TP1	Off
Qwen3.5-27B	BF16	2× S5000 / TP2	MTP1
Qwen3.5-27B	FP8	2× S5000 / TP2	Off
Qwen3.5-35B-A3B	BF16	4× S5000 / TP4	MTP3
Qwen3.5-35B-A3B	FP8	4× S5000 / TP4	MTP3
Qwen3.6-27B	BF16	2× S5000 / TP2	MTP3
Qwen3.6-27B	FP8	2× S5000 / TP2	MTP3
Qwen3.6-35B-A3B	BF16	4× S5000 / TP4	MTP2
Qwen3.6-35B-A3B	FP8	4× S5000 / TP4	MTP2
DeepSeek-V4-Flash-Base	FP8 KV cache	8× S5000 / TP8	Fixed MTP4 (MTP-off available)

Bug Fixes

• Fixed the non-idempotent Qwen3.5 FP8 shared-expert fold, avoiding a duplicate shared route and incorrect results.
• Fixed the DeepSeek-V4-Flash-Base long-prefill indexer memory bounds, TP partitioning, and no-MTP precision semantics; graph execution now handles auxiliary-operation dependencies correctly.
• Fixed the incorrect path caused by registered custom all-reduce inputs in piecewise graphs.
• Fixed the DeepSeek-V4-Flash-Base MTP prefill hang and Qwen3.5 startup with MTP enabled.
• Fixed the Eagle3 Gumbel RNG bitcast and full-graph draft metadata.
• Fixed DeepEP resource cleanup when the EP service exits.
• Fixed the Qwen FP8 graph all-reduce path, MUSA PrivateUse1 repetition-penalty registration, and Triton 3.2 unified-attention decode compatibility.
• Fixed the MATE FA3 API, block-FP8 MoE scale sharding, the obsolete unquantized-MoE argument, and graceful MiniMax RMSNorm degradation without cuda-python.
• Fixed GPTQ and fused activation/layernorm-quant custom-op registration and first-build failures.

Other Changes and Upgrade Notes

• Software stack: use Python 3.10, PyTorch/torch_musa 2.11.0.post1+musa5.2.0, torchada 0.1.83, MATE/flash_attn_3/flash_mla 0.2.4, and Triton 3.2.0. Common dependencies include NumPy 1.26, Transformers 5.5.3, and mthreads-ml-py>=2.4.2.
• Version identifiers: this release line uses upstream vLLM 0.24.0; the vllm-musa plugin Python metadata version is 0.1.24.
• Installation: the MUSA image installs patched vLLM against the existing MUSA PyTorch stack. For standalone wheels, verify the complete dependency set and prevent pip from replacing MUSA PyTorch with a CUDA build.
• Device visibility: set MUSA_VISIBLE_DEVICES explicitly in the deployment environment; the image does not bake in a fixed device list.
• Runtime detection: do not use torch.cuda.is_available() to detect MUSA availability; use the project's MUSA platform helper.

Known Limitations

• Cookbook parameters target an approximately 4K-input / 1K-output bounded-concurrency workload. Re-tune max-num-batched-tokens, max-num-seqs, and graph-capture settings for other request shapes.
• The DeepSeek-V4-Flash-Base MTP-off profile can exceed memory at higher concurrency or on lower-memory deployments. Reduce max-num-seqs and the graph-capture ceiling together before production use.
• FP8, MTP, and CUDA Graph paths are sensitive to the model revision, dependency versions, and input shapes. Validate a real request and semantic output before production deployment.