[Bug]: Block-scaled FP8 (compressed-tensors W8A8) crashes on load on SM120 Blackwell (RTX PRO 6000), v0.24.0 — DeepGEMM "Unknown SF transformation" assertion

[Odrec]

Your current environment

Environment

• vLLM: v0.24.0 (official vllm/vllm-openai:v0.24.0 Docker image)
• GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition — compute capability 12.0 (SM120)
• Driver: 580.126.20
• CUDA (image): 13.0, torch 2.11.0+cu130
• TP: 1 (single GPU)

🐛 Describe the bug

Loading a block-scaled FP8 checkpoint (compressed-tensors, W8A8 FP8 with 2‑D block weight scales) on SM120 Blackwell crashes during weight post-processing. vLLM auto-selects DeepGemmFp8BlockScaledMMKernel, and DeepGEMM's scale-factor layout transform aborts with an assertion:

RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/layout.hpp:59): Unknown SF transformation

Model: RedHatAI/gemma-4-31B-it-FP8-block

Serve command (relevant flags):

vllm serve --model RedHatAI/gemma-4-31B-it-FP8-block \
  --tensor-parallel-size 1 --max-model-len 65536 --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 --async-scheduling --performance-mode interactivity \
  --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 \
  --chat-template examples/tool_chat_template_gemma4.jinja

Kernel selection + traceback:

INFO [__init__.py:563] Selected DeepGemmFp8BlockScaledMMKernel for CompressedTensorsW8A8Fp8
...
ERROR [core.py:1231]   quant_method.process_weights_after_loading(module)
  File ".../compressed_tensors/compressed_tensors.py", line 915, in process_weights_after_loading
    layer.scheme.process_weights_after_loading(layer)
  File ".../compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py", line 169, in process_weights_after_loading
    self.fp8_linear.process_weights_after_loading(layer)
  File ".../kernels/linear/scaled_mm/deep_gemm.py", line 96, in process_weights_after_loading
    dg_weight, dg_weight_scale = deepgemm_post_process_fp8_weight_block(...)
  File ".../quantization/utils/fp8_utils.py", line 1140, in deepgemm_post_process_fp8_weight_block
    dg_ws = deepgemm_post_process_weight_scale_block(...)
  File ".../quantization/utils/fp8_utils.py", line 1079, in deepgemm_post_process_weight_scale_block
    return transform_sf_into_required_layout(...)
  File ".../utils/deep_gemm.py", line 494, in transform_sf_into_required_layout
    return _transform_sf_into_required_layout_impl(...)
RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/layout.hpp:59): Unknown SF transformation

The engine core then fails to initialize (Engine core initialization failed) and the server crash-loops.

Workaround

Disabling DeepGEMM makes it load and serve cleanly — vLLM then selects the Cutlass block-scaled kernel:

VLLM_USE_DEEP_GEMM=0
→ INFO [__init__.py:563] Selected CutlassFp8BlockScaledMMKernel for CompressedTensorsW8A8Fp8
→ INFO: Application startup complete.

So this is specific to the DeepGEMM block-scaled path on SM120, not the checkpoint per se (it also loaded on vLLM v0.20.1).

Suggested fix

Either add the missing SF-layout transform for SM120 in DeepGEMM, or — when DeepGEMM can't handle a checkpoint's scale-factor layout on the current arch — fall back to CutlassFp8BlockScaledMMKernel instead of asserting. (The checkpoint's scale_fmt is not ue8m0, which appears to be what DeepGEMM's Blackwell path requires.)

Related

• [Bug]: Gemma 4 31B FP8_BLOCK checkpoint produces garbage repetitive output — logit saturation at softcap wall due to absorbed activation scales being double-applied #39407 — same checkpoint (gemma-4-31B-it-FP8-block) produces garbage output (double-applied activation scales); different symptom, same checkpoint/area. (Validating whether the Cutlass fallback here is also affected.)
• [Tracking] DeepGEMM SM 12.x kernel coverage gaps for DeepSeek-V4-Flash on consumer Blackwell (RTX 50 / GB10) #41063 — DeepGEMM SM 12.x kernel coverage gaps on consumer Blackwell.
• [Bug] MXFP8 MoE always falls back to MARLIN on SM_121 (DGX Spark / GB10): TrtLlmFp8ExpertsBase gates on family(100), excluding SM_12x consumer Blackwell #43906 — SM_121 MXFP8 MoE fallback gate.

[Odrec]

Follow-up on the VLLM_USE_DEEP_GEMM=0 workaround: with the Cutlass block-scaled kernel the model not only loads but produces healthy output — on GSM8k (n=300, 5-shot) it scored 98.0% (294/300), byte-for-byte identical to the same model's ...-FP8-Dynamic sibling (same 294/300, same 6 misses). So on this eval the Cutlass fallback is not affected by the #39407 garbage-output problem; the DeepGEMM load-crash reported here appears independent of that correctness bug. (RTX PRO 6000 Blackwell, SM120, v0.24.0.)

[waynehacking8]

This is the same v0.24.0 regression tracked in #47130. #43477 added SM120 to CudaPlatform.support_deep_gemm() (vllm/platforms/cuda.py:668 at the v0.24.0 tag), but the DeepGEMM revision v0.24.0 builds against (891d57b4, cmake/external_projects/deepgemm.cmake) predates DeepGEMM's SM120 support: at that revision the SF-layout transform in csrc/apis/layout.hpp has no SM120 case and falls through to DG_HOST_UNREACHABLE("Unknown SF transformation") at layout.hpp:59, the exact line in your traceback.

Your call site differs from #47130 (weight post-processing via DeepGemmFp8BlockScaledMMKernel.process_weights_after_loading, not kernel warmup), which is also why #47258 won't cover it: that fix honors the per-model use_deep_gemm auto-disable (qwen3_5* on SM120) in the warmup and MoE selectors, while gemma-4 compressed-tensors W8A8 still selects the DeepGEMM dense kernel at load.

On current main the composition is coherent again: the pin moved to DeepGEMM nv-dev a6b593d2 ("targeting nv-dev branch due to sm120 support" per the cmake comment), and that revision handles SM100/SM120 in the same transform. The broken pair is specifically v0.24.0's support list plus its DeepGEMM pin. VLLM_USE_DEEP_GEMM=0 is the right interim and matches the CUTLASS fallback suggested in #47130; your GSM8k run on the Cutlass path is good evidence for that route.

I verified the two cmake pins and both layout.hpp revisions (SM120 case absent at 891d57b4, present at a6b593d2); I didn't rerun the failing load myself. Whether 0.24.1 bumps the pin or reverts SM120 from support_deep_gemm() on the release branch is a maintainer call.

[b0bh00d]

Confirming this also happens on DeepSeek-V4-Flash-0731 (native FP8 block-scaled checkpoint, scale_fmt=ue8m0) — not just compressed-tensors W8A8 checkpoints — and that it's still present on the current nightly (not just released versions).

Environment: 2× NVIDIA GB10 (DGX Spark, SM121/12.1), aarch64, 2-node cluster over Ray, --tensor-parallel-size 2, --enable-expert-parallel. Reproduced on vllm==0.26.1rc1.dev248+g55c98e370 (nightly, installed via uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly/cu130 on 2026-08-02), flashinfer-python 0.6.15.post1, torch 2.13.0+cu132.

Identical assertion, one line off from yours (line 60 vs. 59 — presumably just a different commit of the vendored DeepGEMM source):

RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/layout.hpp:60): Unknown SF transformation
  File ".../vllm/model_executor/layers/quantization/fp8.py", line 434, in process_weights_after_loading
    self.fp8_linear.process_weights_after_loading(layer)
  File ".../vllm/model_executor/kernels/linear/scaled_mm/deep_gemm.py", line 96, in process_weights_after_loading
    dg_weight, dg_weight_scale = deepgemm_post_process_fp8_weight_block(...)
  File ".../vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 1137, in deepgemm_post_process_fp8_weight_block
    dg_ws = deepgemm_post_process_weight_scale_block(...)
  File ".../vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 1076, in deepgemm_post_process_weight_scale_block
    return transform_sf_into_required_layout(...)
  File ".../vllm/utils/deep_gemm.py", line 494, in transform_sf_into_required_layout
    return _transform_sf_into_required_layout_impl(...)

Notably this happens on the same launch config that loads and runs fine on stable vllm==0.26.0 (DeepGEMM selected there too — Selected DeepGemmFp8BlockScaledMMKernel for Fp8LinearMethod) — so on this model+hardware combo, nightly is a regression relative to 0.26.0, not a fix. Haven't yet tried the VLLM_USE_DEEP_GEMM=0 workaround from this thread against nightly; will follow up if useful, but flagging in case it saves someone else the round trip given #47818 suggests the Cutlass fallback path may also be broken on SM120 for DeepSeek-V4 specifically.

[meffmadd]

With the latest nightly (nightly-821717118fc26667dd474b9b0ab81d29259dfc5c), I get RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/hyperconnection.hpp:56): Unsupported architecture with VLLM_USE_DEEP_GEMM=0 on RTX PRO 6000 Blackwell.

EDIT: on v0.26.0 without VLLM_USE_DEEP_GEMM=0 I get multiproc_executor.py:1007] tvm.error.InternalError: Check failed: num_tokens > 64 (7 vs. 64) : Decode (num_tokens <= 64) must go through sparse_mla_sm120_decode_dsv3_2 or sparse_mla_sm120_decode_dsv4; got num_tokens=7

[hassellof]

Confirming the fix class from the sglang side: sgl-deep-gemm 0.1.5.post1 (which includes DeepGEMM's SM120 SF-transform support from upstream #324) resolves this assertion family on arch 12 for us in production. Two gotchas if anyone bumps: the 0.1.5.post2 wheel is a source-identical rebuild against torch 2.13 and introduces an earlier dtype assert with older weight pipelines, and the masked FP4 grouped path has a separate uninitialized-padding trap we're reporting to DeepGEMM. Is bumping vLLM's vendored DeepGEMM commit past the SF-transform fix on anyone's radar? The VLLM_USE_DEEP_GEMM=0 fallback gives up real performance.

[khluu]

I opened draft #51382 as a materially different alternative to #50796. Instead of pinning the later nv_dev+situ merge commit 5f33a180, it pins 586032e, whose parent is exactly the previously used SM120 commit a6b593d with only the f5a7642 SITU change cherry-picked on top. GitHub reports the two candidate DeepGEMM revisions are diverged and differ across 79 files (10 commits ahead / 1 behind), so the draft intentionally minimizes the dependency delta. Static checks passed, but CUDA/SM120 and model validation are still required before it should be marked ready.

[ThiemoClausen]

I updated to latest nightly and also get this with deepseek v4 flash on dual RTX 6000:

 WorkerProc failed to start.
 Traceback (most recent call last):
   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 885, in worker_main
     worker = WorkerProc(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
     return func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 654, in __init__
     self.worker.load_model()
   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 443, in load_model
     self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
     return func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5403, in load_model
     self.model = model_loader.load_model(
                  ^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
     return func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 80, in load_model
     process_weights_after_loading(model, model_config, target_device)
   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/utils.py", line 111, in process_weights_after_loading
     quant_method.process_weights_after_loading(module)
   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/fp8.py", line 411, in process_weights_after_loading
     self.fp8_linear.process_weights_after_loading(layer)
   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/scaled_mm/deep_gemm.py", line 96, in process_weights_after_loading
     dg_weight, dg_weight_scale = deepgemm_post_process_fp8_weight_block(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 1138, in deepgemm_post_process_fp8_weight_block
     dg_ws = deepgemm_post_process_weight_scale_block(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 1077, in deepgemm_post_process_weight_scale_block
     return transform_sf_into_required_layout(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/deep_gemm.py", line 494, in transform_sf_into_required_layout
     return _transform_sf_into_required_layout_impl(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/layout.hpp:60): Unknown SF transformation
[shutdown] Executor: waiting for worker exit count=2