[Bug]: vLLM does not support DeepSeek series on RTX PRO 6000/SM120

[yuanyuan19]

Your current environment

Here is my configuration information:

PRETTY_NAME="Ubuntu 24.04.3 LTS"
=== GPU Information ===
index, name, compute_cap, memory.total [MiB], driver_version
0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 12.0, 97887 MiB, 580.65.06
1, NVIDIA RTX PRO 6000 Blackwell Server Edition, 12.0, 97887 MiB, 580.65.06
2, NVIDIA RTX PRO 6000 Blackwell Server Edition, 12.0, 97887 MiB, 580.65.06
3, NVIDIA RTX PRO 6000 Blackwell Server Edition, 12.0, 97887 MiB, 580.65.06
4, NVIDIA RTX PRO 6000 Blackwell Server Edition, 12.0, 97887 MiB, 580.65.06
5, NVIDIA RTX PRO 6000 Blackwell Server Edition, 12.0, 97887 MiB, 580.65.06
6, NVIDIA RTX PRO 6000 Blackwell Server Edition, 12.0, 97887 MiB, 580.65.06
7, NVIDIA RTX PRO 6000 Blackwell Server Edition, 12.0, 97887 MiB, 580.65.06
8, NVIDIA RTX PRO 6000 Blackwell Server Edition, 12.0, 97887 MiB, 580.65.06

CUDA: 12.9
PyTorch uses https://download.pytorch.org/whl/nightly/cu130, but version 12.8 has also been tried.

🐛 Describe the bug

Reproducing the issue on SM120:

llm = LLM(
    model="deepseek-ai/DeepSeek-V3.1",     # Same applies to deepseek-ai/DeepSeek-V3.2-Exp
    tensor_parallel_size=8,
    max_model_len=4096,
    gpu_memory_utilization=0.5, 
)

Throws an exception: Invalid status

RuntimeError: Invalid status
# At line 667 in vllm/_custom_ops.py
torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
# RuntimeError: Invalid status is thrown here

Here are the issues I've collected, which might serve as references:
Running DeepSeek R1 on 8x RTX 6000 PRO · Issue #5581 · NVIDIA/TensorRT-LLM

According to this issue, it seems that the FP8 block-scaled GEMM kernel written for SM100 does not work on RTX Pro 6000 (SM120) because SM120 is not a superset of the SM100 architecture and requires a new kernel.

This is too complex for me. I hope someone can resolve this issue and contribute to the vLLM community ecosystem.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ZJY0516]

I think vLLM has cutlass_scaled_mm for sm120

https://github.com/vllm-project/vllm/blob/main/csrc%2Fquantization%2Fcutlass_w8a8%2Fscaled_mm_entry.cu#L175-L175

[yuanyuan19]

@ZJY0516 I apologize for the misunderstanding caused by my description. This was Cursor's response after I provided it with the runtime error:

DeepSeek-V3.2-Exp-FP8 Model Loading
    ↓
FP8 Quantization Layer Initialization
    ↓
cutlass_scaled_mm Call
    ↓
torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
    ↓
cutlass_scaled_mm() [scaled_mm_entry.cu:175]
    ↓
cutlass_scaled_mm_sm120() [For SM 12.0]
    ↓
CUTLASS_CHECK(status) [common.hpp:11]
    ↓
TORCH_CHECK(error == cutlass::Status::kSuccess, "Invalid status")
    ↓
RuntimeError: Invalid status

A more detailed process, starting from inside cutlass_scaled_mm_sm120():

// At vllm/csrc/quantization/cutlass_w8a8/c3x/scaled_mm_sm120_fp8_dispatch.cuh line 31
inline void cutlass_gemm_sm120_fp8_dispatch(torch::Tensor& out,
                                            torch::Tensor const& a,
                                            torch::Tensor const& b,
                                            EpilogueArgs&&... args) {
  // ... Configuration selection ...
  // Lines 42-43: Call CUTLASS GEMM caller
  return cutlass_gemm_caller<Cutlass3xGemmDefault>(
      out, a, b, std::forward<EpilogueArgs>(args)...);
}

// vllm/csrc/quantization/cutlass_w8a8/c3x/cutlass_gemm_caller.cuh:34

void cutlass_gemm_caller(
    torch::Device device, cute::Shape<int, int, int, int> prob_shape,
    typename GemmKernel::MainloopArguments mainloop_args,
    typename GemmKernel::EpilogueArguments epilogue_args,
    typename GemmKernel::TileSchedulerArguments scheduler = {}) {
  // ... Parameter preparation ...
  
  // Line 50: Check if implementation is possible
  CUTLASS_CHECK(gemm_op.can_implement(args));
  
  // ... Workspace allocation ...
  
  // Lines 59-60: 🚨 Where the error occurs!
  cutlass::Status status = gemm_op.run(args, workspace.data_ptr(), stream);
  CUTLASS_CHECK(status);  // ← This throws the "Invalid status" error
}

This is the complete error:

vllm serve deepseek-ai/DeepSeek-V3.2-Exp -tp 8 --gpu-memory-utilization 0.7

bug.txt

This is too complex for me. I hope someone capable can help.

[ZJY0516]

I don't have sm120 machine to reproduce this. Do you have any insights or suggestions here? @yewentao256 @LucasWilkinson

[yewentao256]

Sorry, I am worried we don't have SM120 as well currently. @simon-mo Wondering do we have dev machine for SM120?

[simon-mo]

We have rtx6000 pro. Let me add you

[sirus20x6]

I have the same issue. I think I must have recompiled more than a dozen times by now trying to fix it

File "/usr/lib/python3.13/site-packages/vllm/entrypoints/cli/benchmark/latency.py", line 5, in <module> from vllm.benchmarks.latency import add_cli_args, main File "/usr/lib/python3.13/site-packages/vllm/benchmarks/latency.py", line 18, in <module> from vllm.engine.arg_utils import EngineArgs File "/usr/lib/python3.13/site-packages/vllm/engine/arg_utils.py", line 24, in <module> from vllm.config import (BlockSize, CacheConfig, CacheDType, CompilationConfig, ...<9 lines>... VllmConfig, get_attr_docs) File "/usr/lib/python3.13/site-packages/vllm/config/__init__.py", line 26, in <module> from vllm.config.cache import (BlockSize, CacheConfig, CacheDType, MambaDType, PrefixCachingHashAlgo) File "/usr/lib/python3.13/site-packages/vllm/config/cache.py", line 15, in <module> from vllm.utils import GiB_bytes, get_cpu_memory File "/usr/lib/python3.13/site-packages/vllm/utils/__init__.py", line 56, in <module> import cbor2 ModuleNotFoundError: No module named 'cbor2' ❯ source ./venv/bin/activate ❯ vllm serve \ /thearray/git/ob/text-generation-webui/models/Qwen3-VL-30B-A3B-Instruct-AWQ \ --served-model-name Qwen3-VL \ --max-model-len 65536 \ --gpu-memory-utilization 0.95 \ --trust-remote-code \ --port 9900 \ --disable-log-stats /thearray/git/vllm/venv/lib/python3.13/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did n import pynvml # type: ignore[import] INFO 10-05 21:08:54 [__init__.py:224] Automatically detected platform cuda. (APIServer pid=2170777) INFO 10-05 21:09:00 [api_server.py:1882] vLLM API server version 0.11.0rc2.dev230+gf509a2084.d20251006 (APIServer pid=2170777) INFO 10-05 21:09:00 [utils.py:239] non-default args: {'model_tag': '/thearray/git/ob/text-generation-webui/models/Qwen3-VL-30B-A3B-Instruct-AWQ', 'port': (APIServer pid=2170777) INFO 10-05 21:09:00 [model.py:597] Resolved architecture: Qwen3VLMoeForConditionalGeneration (APIServer pid=2170777) torch_dtypeis deprecated! Usedtypeinstead! (APIServer pid=2170777) INFO 10-05 21:09:00 [model.py:1658] Using max model len 65536 (APIServer pid=2170777) INFO 10-05 21:09:00 [awq_marlin.py:154] The model is convertible to awq_marlin during runtime. Using awq_marlin kernel. (APIServer pid=2170777) The argumenttrust_remote_code is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=2170777) INFO 10-05 21:09:01 [scheduler.py:225] Chunked prefill is enabled with max_num_batched_tokens=8192. /thearray/git/vllm/venv/lib/python3.13/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did n import pynvml # type: ignore[import] INFO 10-05 21:09:08 [__init__.py:224] Automatically detected platform cuda. (EngineCore_DP0 pid=2170889) INFO 10-05 21:09:14 [core.py:715] Waiting for init message from front-end. (EngineCore_DP0 pid=2170889) INFO 10-05 21:09:14 [core.py:94] Initializing a V1 LLM engine (v0.11.0rc2.dev230+gf509a2084.d20251006) with config: model='/thearray/git/ob/text-gene [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 (EngineCore_DP0 pid=2170889) INFO 10-05 21:09:16 [parallel_state.py:1231] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 (EngineCore_DP0 pid=2170889) INFO 10-05 21:09:16 [topk_topp_sampler.py:59] Using FlashInfer for top-p & top-k sampling. (EngineCore_DP0 pid=2170889) INFO 10-05 21:09:22 [gpu_model_runner.py:2904] Starting to load model /thearray/git/ob/text-generation-webui/models/Qwen3-VL-30B-A3B-Instruct-AWQ... (EngineCore_DP0 pid=2170889) INFO 10-05 21:09:22 [gpu_model_runner.py:2934] Loading model from scratch... (EngineCore_DP0 pid=2170889) INFO 10-05 21:09:22 [cuda.py:392] Using Flash Attention backend on V1 engine. Loading safetensors checkpoint shards: 0% Completed | 0/6 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 17% Completed | 1/6 [00:03<00:18, 3.71s/it] Loading safetensors checkpoint shards: 33% Completed | 2/6 [00:07<00:15, 3.83s/it] Loading safetensors checkpoint shards: 50% Completed | 3/6 [00:11<00:11, 3.88s/it] Loading safetensors checkpoint shards: 67% Completed | 4/6 [00:15<00:07, 3.85s/it] Loading safetensors checkpoint shards: 83% Completed | 5/6 [00:17<00:03, 3.19s/it] Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:20<00:00, 3.17s/it] Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:20<00:00, 3.42s/it] (EngineCore_DP0 pid=2170889) (EngineCore_DP0 pid=2170889) INFO 10-05 21:09:43 [default_loader.py:309] Loading weights took 20.74 seconds (EngineCore_DP0 pid=2170889) INFO 10-05 21:09:59 [gpu_model_runner.py:2957] Model loading took 17.0288 GiB and 36.761308 seconds (EngineCore_DP0 pid=2170889) INFO 10-05 21:09:59 [gpu_model_runner.py:3682] Encoder cache will be initialized with a budget of 153600 tokens, and profiled with 1 video items of t Invalid handle. Cannot load symbol cublasLtCreate (APIServer pid=2170777) Traceback (most recent call last): (APIServer pid=2170777) File "/thearray/git/vllm/venv/bin/vllm", line 10, in <module> (APIServer pid=2170777) sys.exit(main()) (APIServer pid=2170777) ~~~~^^ (APIServer pid=2170777) File "/thearray/git/vllm/vllm/entrypoints/cli/main.py", line 54, in main (APIServer pid=2170777) args.dispatch_function(args) (APIServer pid=2170777) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^ (APIServer pid=2170777) File "/thearray/git/vllm/vllm/entrypoints/cli/serve.py", line 63, in cmd (APIServer pid=2170777) uvloop.run(run_server(args)) (APIServer pid=2170777) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) File "/thearray/git/vllm/venv/lib/python3.13/site-packages/uvloop/__init__.py", line 109, in run (APIServer pid=2170777) return __asyncio.run( (APIServer pid=2170777) ~~~~~~~~~~~~~^ (APIServer pid=2170777) wrapper(), (APIServer pid=2170777) ^^^^^^^^^^ (APIServer pid=2170777) ...<2 lines>... (APIServer pid=2170777) **run_kwargs (APIServer pid=2170777) ^^^^^^^^^^^^ (APIServer pid=2170777) ) (APIServer pid=2170777) ^ (APIServer pid=2170777) File "/usr/lib/python3.13/asyncio/runners.py", line 195, in run (APIServer pid=2170777) return runner.run(main) (APIServer pid=2170777) ~~~~~~~~~~^^^^^^ (APIServer pid=2170777) File "/usr/lib/python3.13/asyncio/runners.py", line 118, in run (APIServer pid=2170777) return self._loop.run_until_complete(task) (APIServer pid=2170777) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ (APIServer pid=2170777) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete (APIServer pid=2170777) File "/thearray/git/vllm/venv/lib/python3.13/site-packages/uvloop/__init__.py", line 61, in wrapper (APIServer pid=2170777) return await main (APIServer pid=2170777) ^^^^^^^^^^ (APIServer pid=2170777) File "/thearray/git/vllm/vllm/entrypoints/openai/api_server.py", line 1926, in run_server (APIServer pid=2170777) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) (APIServer pid=2170777) File "/thearray/git/vllm/vllm/entrypoints/openai/api_server.py", line 1942, in run_server_worker (APIServer pid=2170777) async with build_async_engine_client( (APIServer pid=2170777) ~~~~~~~~~~~~~~~~~~~~~~~~~^ (APIServer pid=2170777) args, (APIServer pid=2170777) ^^^^^ (APIServer pid=2170777) client_config=client_config, (APIServer pid=2170777) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) ) as engine_client: (APIServer pid=2170777) ^ (APIServer pid=2170777) File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ (APIServer pid=2170777) return await anext(self.gen) (APIServer pid=2170777) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) File "/thearray/git/vllm/vllm/entrypoints/openai/api_server.py", line 191, in build_async_engine_client (APIServer pid=2170777) async with build_async_engine_client_from_engine_args( (APIServer pid=2170777) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ (APIServer pid=2170777) engine_args, (APIServer pid=2170777) ^^^^^^^^^^^^ (APIServer pid=2170777) ...<2 lines>... (APIServer pid=2170777) client_config=client_config, (APIServer pid=2170777) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) ) as engine: (APIServer pid=2170777) ^ (APIServer pid=2170777) File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ (APIServer pid=2170777) return await anext(self.gen) (APIServer pid=2170777) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) File "/thearray/git/vllm/vllm/entrypoints/openai/api_server.py", line 238, in build_async_engine_client_from_engine_args (APIServer pid=2170777) async_llm = AsyncLLM.from_vllm_config( (APIServer pid=2170777) vllm_config=vllm_config, (APIServer pid=2170777) ...<5 lines>... (APIServer pid=2170777) client_index=client_index, (APIServer pid=2170777) ) (APIServer pid=2170777) File "/thearray/git/vllm/vllm/utils/__init__.py", line 1590, in inner (APIServer pid=2170777) return fn(*args, **kwargs) (APIServer pid=2170777) File "/thearray/git/vllm/vllm/v1/engine/async_llm.py", line 209, in from_vllm_config (APIServer pid=2170777) return cls( (APIServer pid=2170777) vllm_config=vllm_config, (APIServer pid=2170777) ...<8 lines>... (APIServer pid=2170777) client_index=client_index, (APIServer pid=2170777) ) (APIServer pid=2170777) File "/thearray/git/vllm/vllm/v1/engine/async_llm.py", line 133, in __init__ (APIServer pid=2170777) self.engine_core = EngineCoreClient.make_async_mp_client( (APIServer pid=2170777) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ (APIServer pid=2170777) vllm_config=vllm_config, (APIServer pid=2170777) ^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) ...<4 lines>... (APIServer pid=2170777) client_index=client_index, (APIServer pid=2170777) ^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) ) (APIServer pid=2170777) ^ (APIServer pid=2170777) File "/thearray/git/vllm/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client (APIServer pid=2170777) return AsyncMPClient(*client_args) (APIServer pid=2170777) File "/thearray/git/vllm/vllm/v1/engine/core_client.py", line 809, in __init__ (APIServer pid=2170777) super().__init__( (APIServer pid=2170777) ~~~~~~~~~~~~~~~~^ (APIServer pid=2170777) asyncio_mode=True, (APIServer pid=2170777) ^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) ...<3 lines>... (APIServer pid=2170777) client_addresses=client_addresses, (APIServer pid=2170777) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) ) (APIServer pid=2170777) ^ (APIServer pid=2170777) File "/thearray/git/vllm/vllm/v1/engine/core_client.py", line 470, in __init__ (APIServer pid=2170777) with launch_core_engines(vllm_config, executor_class, log_stats) as ( (APIServer pid=2170777) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) File "/usr/lib/python3.13/contextlib.py", line 148, in __exit__ (APIServer pid=2170777) next(self.gen) (APIServer pid=2170777) ~~~~^^^^^^^^^^ (APIServer pid=2170777) File "/thearray/git/vllm/vllm/v1/engine/utils.py", line 773, in launch_core_engines (APIServer pid=2170777) wait_for_engine_startup( (APIServer pid=2170777) ~~~~~~~~~~~~~~~~~~~~~~~^ (APIServer pid=2170777) handshake_socket, (APIServer pid=2170777) ^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) ...<5 lines>... (APIServer pid=2170777) coordinator.proc if coordinator else None, (APIServer pid=2170777) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=2170777) ) (APIServer pid=2170777) ^ (APIServer pid=2170777) File "/thearray/git/vllm/vllm/v1/engine/utils.py", line 830, in wait_for_engine_startup (APIServer pid=2170777) raise RuntimeError( (APIServer pid=2170777) ...<3 lines>... (APIServer pid=2170777) ) (APIServer pid=2170777) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} /usr/lib/python3.13/multiprocessing/resource_tracker.py:324: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {'/mp-4qlt3jco'} warnings.warn(

[sirus20x6]

"According to this issue, it seems that the FP8 block-scaled GEMM kernel written for SM100 does not work on RTX Pro 6000 (SM120) because SM120 is not a superset of the SM100 architecture and requires a new kernel"

This is correct. In fact I'm a little pissed off at the things SM120 is missing. They are scalling up the consumer side instead of scaling down the extreme enterprise side for their workstation cards.

[yuanyuan19]

I can provide a machine capable of reproducing the issue, if you still need it. @ZJY0516 @yewentao256

[sirus20x6]

i might be able to provide a machine. depending on what's required

[yewentao256]

@yuanyuan19 Sounds great, the node we have just have two cards..
@ZJY0516 Could you use @yuanyuan19 's node and see whether we can reproduce the issue?

[ZJY0516]

@yuanyuan19 Sounds great, the node we have just have two cards.. @ZJY0516 Could you use @yuanyuan19 's node and see whether we can reproduce the issue?

Yes, but honestly I am not very familiar with cutlass.
BTW, you can try DeepSeek-V2-Lite which has similar architecture with DeepSeek V3

[yuanyuan19]

Could you please leave an email address? I'll send you the SSH account and password. You can freely experiment with this server, as deepseek-v3.2 has been downloaded locally. @ZJY0516 @yewentao256 . If it can't be resolved, it's not a big deal.

[ZJY0516]

@yuanyuan19 riverclouds.zhu@qq.com Thanks!

[yuanyuan19]

@ZJY0516 Sent. Please reply here once you’ve received the email. This server is currently unused. Let me know if you no longer need it. All the best.