[Performance]: Deepseek-V4 Support and Optimization on ROCm Backend

[wuhuikx]

Motivation

This issue tracks the end-to-end enablement and optimization checklist for DeepSeek-V4 on ROCm backend.

DeepSeek-V4 includes multiple critical blocks (mHC/HCA/CSA/MoE/MTP), and ROCm readiness depends on both model-side kernels and system-side runtime behavior.

We’re launching a joint effort to optimize DeepSeek V4 on the ROCm backend—please feel free to take on any task, and we’d love to hear more optimization ideas.

Purpose

• Track DeepSeek-V4 functionality and performance readiness on ROCm backend.
• Keep module-level optimization items visible and actionable.
• Align acceptance criteria for release and production readiness.

---

Recipe

• Recipe: vLLM Recipe: DeepSeek-V4 on AMD (ROCm) Usage Guide - vLLM Recipes [ROCm] Add the Deepseek-V4-Pro supported on MI355x recipes#433

General Checklist

1) Functionality / Bugfix / Feature

• [Functionality] Base PR has been merged for functionality/accuracy readiness on MI35x for DeepSeek-V4-Pro and DeepSeek-V4-Flash. [New Model][ROCm] Add AMD support for DeepSeek V4 #40871
• [Functionality] DeepSeek-V4-Flash Base FP8 enablement PR: [ROCm] DeepSeekV4-Flash-Base model enablement on ROCm with triton & torchfallback #41136
• [Functionality] MI30x support PR: [ROCm][Deepseekv4] DeepseekV4 Mi300 support  #41451
• [Bugfix] Enable cross-node TP=16 FP8 serving for DeepSeek-V4: [Bugfix][DeepSeek V4] Enable cross-node TP=16 FP8 serving #41312
• [Feature][WIP] Enable KV offload for DeepSeek-V4: [feature][WIP] Enable KV Offload for DeepSeek V4 model #41352
• [Feature][WIP] Compressed tensors support for DSV4: [CompressedTensors] DeepSeek4 CT Quantization Support #41276
• [Bugfix] Fix accuracy issue with high concurrency: [ROCm] [Bugfix] Fix DeepSeek V4 Functionality and Accuracy #42810

---

Performance Checklist

1) High-Level Performance/Feature

• Full graph support for Decode path. Piecewise graph capture support for Prefill path. DeepSeekV4-Pro enable cuda graph full and piecewise mode #42604 DeepSeekv4 ROCm Optimization #41601
• Support MTP

2) Kernel Fusion

Element-wise Fusion

• Avoid redundant dtype conversion: [DSV4] Avoid redundant dtype conversion. #41374
• Fuse norm and router for lower latency: [DSV4] Fuse norm and router for low latency scenario #41263
• Silu and Mul is implemented with torch native
• inv_rope is implemented with torch native

CSA

• Make the sparse MLA indexer optimization for Deepseek-V4, now the sparse attention indexder is pytorch native implementation.
• Replace torch native sparse MLA path with Triton kernel ([ROCm] DeepSeekV4-Flash-Base model enablement on ROCm with triton & torchfallback #41136). Another single kernel ([ROCm][DSv4] implement flash sparse mla with triton kernels #41812)
• Enable CSA multi-stream execution in Decode (default stream + indexer stream) to overlap indexer and main attention paths, aligned with the DeepSeek-V4 blog design.

mHC

• Now the MHC on ROCm is pytorch native implementation (mhc_pre， mhc_post), need to integrate AITER MHC kernel [Bugfix] [ROCm] [DSV4] [Perf] Add aiter mhc support #41946

MoE

• AITER FlyDSL MoE integration when it's ready (MoE Kernel)

[github-actions]

CC @hongxiayang @tjtanaa @vllmellm for ROCm-related issue

[wuhuikx]

Hi @tjtanaa @ganyi1996ppo @whx-sjtu the graph and MLA PRs are very important, could you prioritize to review them? #41601 and #41136

[tjtanaa]

I can help with mhc integration.

[bobofang11235]

inv_rope related PR:

• aiter: Add fused inv_rope + FP8 block-scaled quantization kernel for DSV4 MLA ROCm/aiter#3079
• vllm: Inv rope fp8 quant #42033 ( based on DeepSeekv4 ROCm Optimization #41601 )

Update the test result in vllm PR

[qli88]

I will help to implement "Silu and Mul" with Triton. @tjtanaa

[Hanzamas]

sir please sir i need this xD

[tjtanaa]

I am working on fp4 indexer and MTP enablement.

Finalizing
fp4Indexer PR: #42908
MTP PR #43385