Your current environment
Environment
- GPU: NVIDIA GeForce RTX 5090
- Compute capability: 12.0
- VRAM: 32607 MiB
- Driver: 610.43.02
- CUDA UMD: 13.3
- OS: Arch Linux, GNOME desktop
- vLLM: 0.24.0
- PyTorch: 2.11.0+cu130
- flashinfer-python: 0.6.12
- Python: 3.11.15
- Model:
nvidia/Qwen3.6-27B-NVFP4
nvidia-smi:
NVIDIA GeForce RTX 5090, 12.0, 610.43.02, 32607 MiB
NVIDIA-SMI 610.43.02
CUDA UMD Version: 13.3
🐛 Describe the bug
Summary
nvidia/Qwen3.6-27B-NVFP4 on RTX 5090 / SM120 resolves as modelopt_mixed, detects W4A16_NVFP4, then
routes part of the model through the Marlin FP4 path and logs that the GPU does not have native FP4
support.
The hardware does support FP4: this is an RTX 5090 Blackwell GPU with compute capability 12.0. The issue
appears to be vLLM backend selection / support for ModelOpt mixed NVFP4 checkpoints on SM120, or a
misleading capability warning from the Marlin fallback path.
Command
vllm serve nvidia/Qwen3.6-27B-NVFP4
--host 0.0.0.0
--port 8082
--tensor-parallel-size 1
--load-format safetensors
--attention-backend flashinfer
--quantization modelopt_fp4
--dtype bfloat16
--kv-cache-dtype fp8_e4m3
--max-model-len 32768
--max-num-seqs 1
--max-num-batched-tokens 8192
--gpu-memory-utilization 0.88
--enable-chunked-prefill
--compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1]}'
--no-enable-flashinfer-autotune
--skip-mm-profiling
--reasoning-parser qwen3
--tool-call-parser qwen3_coder
--enable-auto-tool-choice
Relevant Logs
Resolved architecture: Qwen3_5ForConditionalGeneration
Using max model len 32768
Chunked prefill is enabled with max_num_batched_tokens=8192.
Detected ModelOpt fp8 checkpoint (quant_algo=FP8).
Detected ModelOpt NVFP4 checkpoint (quant_algo=NVFP4).
Detected ModelOpt NVFP4 checkpoint (quant_algo=W4A16_NVFP4).
Initializing a V1 LLM engine ... quantization=modelopt_mixed ... device_config=cuda
Using FlashInfer for top-p & top-k sampling.
Selected FlashInferFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod
Using AttentionBackendEnum.FLASHINFER backend.
WARNING [marlin.py:34] Your GPU does not have native support for FP4 computation but FP4 quantization is
being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade
performance for compute-heavy workloads.
Observed Behavior
vLLM starts and serves the model, but prompt processing and token generation are much slower than expected
for a native-FP4-capable Blackwell GPU.
The concerning part is the Marlin warning saying the GPU lacks native FP4 support, despite the GPU being
an RTX 5090 / SM120. From inspecting vllm/model_executor/layers/quantization/modelopt.py in vLLM 0.24.0,
the W4A16_NVFP4 path appears to instantiate MarlinNvFp4LinearKernel directly, so this may be a backend-
selection/support gap rather than actual hardware detection failure.
Expected Behavior
On SM120 / RTX 5090, ModelOpt NVFP4 layers should use native Blackwell FP4-capable kernels where
supported, or the warning should clearly state that only this specific W4A16_NVFP4 checkpoint path
currently falls back to Marlin.
If W4A16_NVFP4 native execution is not currently supported on SM120, it would be helpful for the logs/docs
to make that explicit, because the current warning implies the GPU itself lacks native FP4 support.
Additional Notes
I also saw repeated FlashInfer autotune OOM fallback warnings during startup before disabling autotune:
[Autotuner]: OOM detected, falling back to default tactic
Disabling autotune reduced that noise/startup pressure, but does not address the Marlin FP4 fallback
warning.
Related issues I found, but none seemed to exactly cover this single-GPU RTX 5090 + Qwen3.6-27B-NVFP4 +
modelopt_mixed + W4A16_NVFP4 Marlin fallback case:
Before submitting a new issue...
Your current environment
Environment
nvidia/Qwen3.6-27B-NVFP4nvidia-smi:🐛 Describe the bug
Summary
nvidia/Qwen3.6-27B-NVFP4on RTX 5090 / SM120 resolves asmodelopt_mixed, detectsW4A16_NVFP4, thenroutes part of the model through the Marlin FP4 path and logs that the GPU does not have native FP4
support.
The hardware does support FP4: this is an RTX 5090 Blackwell GPU with compute capability 12.0. The issue
appears to be vLLM backend selection / support for ModelOpt mixed NVFP4 checkpoints on SM120, or a
misleading capability warning from the Marlin fallback path.
Command
vllm serve nvidia/Qwen3.6-27B-NVFP4
--host 0.0.0.0
--port 8082
--tensor-parallel-size 1
--load-format safetensors
--attention-backend flashinfer
--quantization modelopt_fp4
--dtype bfloat16
--kv-cache-dtype fp8_e4m3
--max-model-len 32768
--max-num-seqs 1
--max-num-batched-tokens 8192
--gpu-memory-utilization 0.88
--enable-chunked-prefill
--compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1]}'
--no-enable-flashinfer-autotune
--skip-mm-profiling
--reasoning-parser qwen3
--tool-call-parser qwen3_coder
--enable-auto-tool-choice
Relevant Logs
Resolved architecture: Qwen3_5ForConditionalGeneration
Using max model len 32768
Chunked prefill is enabled with max_num_batched_tokens=8192.
Detected ModelOpt fp8 checkpoint (quant_algo=FP8).
Detected ModelOpt NVFP4 checkpoint (quant_algo=NVFP4).
Detected ModelOpt NVFP4 checkpoint (quant_algo=W4A16_NVFP4).
Initializing a V1 LLM engine ... quantization=modelopt_mixed ... device_config=cuda
Using FlashInfer for top-p & top-k sampling.
Selected FlashInferFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod
Using AttentionBackendEnum.FLASHINFER backend.
WARNING [marlin.py:34] Your GPU does not have native support for FP4 computation but FP4 quantization is
being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade
performance for compute-heavy workloads.
Observed Behavior
vLLM starts and serves the model, but prompt processing and token generation are much slower than expected
for a native-FP4-capable Blackwell GPU.
The concerning part is the Marlin warning saying the GPU lacks native FP4 support, despite the GPU being
an RTX 5090 / SM120. From inspecting vllm/model_executor/layers/quantization/modelopt.py in vLLM 0.24.0,
the W4A16_NVFP4 path appears to instantiate MarlinNvFp4LinearKernel directly, so this may be a backend-
selection/support gap rather than actual hardware detection failure.
Expected Behavior
On SM120 / RTX 5090, ModelOpt NVFP4 layers should use native Blackwell FP4-capable kernels where
supported, or the warning should clearly state that only this specific W4A16_NVFP4 checkpoint path
currently falls back to Marlin.
If W4A16_NVFP4 native execution is not currently supported on SM120, it would be helpful for the logs/docs
to make that explicit, because the current warning implies the GPU itself lacks native FP4 support.
Additional Notes
I also saw repeated FlashInfer autotune OOM fallback warnings during startup before disabling autotune:
[Autotuner]: OOM detected, falling back to default tactic
Disabling autotune reduced that noise/startup pressure, but does not address the Marlin FP4 fallback
warning.
Related issues I found, but none seemed to exactly cover this single-GPU RTX 5090 + Qwen3.6-27B-NVFP4 +
modelopt_mixed + W4A16_NVFP4 Marlin fallback case:
Before submitting a new issue...