[Bug]: [Fp8] [MoE] 'FLASHINFER_CUTLASS' is auto-selected as MoE backend instead of 'DEEPGEMM' on hopper

[cjackal]

Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

🐛 Describe the bug

On H100 x 8 worker and using the latest vllm (here I am using vllm==0.15.2rc1.dev152+g9608844f9), launching a non-mla fp8 MoE model server prints the following log at the beginning:

example vllm launch script

Here I use Qwen3MoE checkpoint, but theoretically any non-mla fp8 MoE checkpoints would suffer similar error.

vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 \
  --gpu-utilization 0.95 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --max-model-len 262144 \
  --max-num-batched-tokens 16384  # set --max-num-batched-tokens to prove that flashinfer fp8 does not work over this token limit

FP8 MoE backend selection log

...
(Worker_TP0_EP0 pid=955) INFO 02-10 20:34:43 [fp8.py:329] Using FLASHINFER_CUTLASS Fp8 MoE backend out of potential backends: ['AITER', 'FLASHINFER_TRTLLM', 'FLASHINFER_CUTLASS', 'DEEPGEMM', 'BATCHED_DEEPGEMM', 'TRITON', 'BATCHED_TRITON', 'MARLIN'].
...

Error traceback for >16k input

With the launch script above, the model server launches successfully and works well for short (<16k here, --max-num-batched-tokens is the threshold) inputs. But if a long request is sent, the following error is raised:

The error log - folded due to its length

(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] WorkerProc hit an exception.
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] Traceback (most recent call last):
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 858, in worker_busy_loop
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     output = func(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 361, in execute_model
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.worker.execute_model(scheduler_output)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return func(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 652, in execute_model
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     output = self.model_runner.execute_model(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return func(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3538, in execute_model
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     model_output = self._model_forward(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]                    ^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3047, in _model_forward
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.model(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 222, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.runnable(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self._call_impl(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return forward_call(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 784, in forward
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     hidden_states = self.model(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]                     ^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 471, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return TorchCompileWithNoGuardsWrapper.__call__(self, *args, **kwargs)  # type: ignore[arg-type]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/wrapper.py", line 225, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self._call_with_optional_nvtx_range(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/wrapper.py", line 119, in _call_with_optional_nvtx_range
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return callable_fn(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 471, in forward
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     def forward(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1181, in _fn
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return fn(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/caching.py", line 185, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.optimized_call(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 936, in call_wrapped
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 455, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     raise e
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 442, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self._call_impl(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return forward_call(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "<eval_with_key>.195", line 864, in forward
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     submod_2 = self.submod_2(getitem_3, s72, getitem_4, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_parameters_weight_scale_inv_, l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_, getitem_5, l_self_modules_layers_modules_0_modules_mlp_modules_gate_parameters_weight_, l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_weight_scale_inv_, l_self_modules_layers_modules_1_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  getitem_3 = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_parameters_weight_scale_inv_ = l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_ = getitem_5 = l_self_modules_layers_modules_0_modules_mlp_modules_gate_parameters_weight_ = l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_weight_scale_inv_ = l_self_modules_layers_modules_1_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_k_norm_parameters_weight_ = None
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/cuda_graph.py", line 222, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.runnable(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/piecewise_backend.py", line 333, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return range_entry.runnable(*args)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_inductor/standalone_compile.py", line 122, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self._compiled_fn(*args)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1181, in _fn
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return fn(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1148, in forward
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return compiled_fn(full_args)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 357, in runtime_wrapper
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     all_outs = call_func_at_runtime_with_args(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 134, in call_func_at_runtime_with_args
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     out = normalize_as_list(f(args))
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]                             ^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1962, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.compiled_fn(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 531, in wrapper
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return compiled_fn(runtime_args)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 729, in inner_fn
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     outs = compiled_fn(args)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 638, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.current_callable(inputs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_inductor/utils.py", line 3220, in run
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     out = model(new_inputs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]           ^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/tmp/cache/torchinductor_root/ab/cabfxt2i72its57ah57fl57sx5xry5ycqhhnwahhvxhcgtyusna5.py", line 1128, in call
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     buf13 = torch.ops.vllm.moe_forward.default(buf11, buf12, 'from_forward_context')
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/_ops.py", line 819, in __call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self._op(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/layer.py", line 2068, in moe_forward
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.forward_impl(hidden_states, router_logits)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/layer.py", line 1936, in forward_impl
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     final_hidden_states = self.quant_method.apply(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]                           ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/fp8.py", line 1016, in apply
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.moe_mk(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self._call_impl(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return forward_call(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1367, in forward
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     fused_out = self._fused_experts(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]                 ^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1203, in _fused_experts
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     self.fused_experts.apply(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/flashinfer_cutlass_moe.py", line 273, in apply
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     _ = flashinfer_cutlass_fused_moe(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/utils/flashinfer.py", line 102, in wrapper
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return impl(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/flashinfer/fused_moe/core.py", line 893, in cutlass_fused_moe
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return get_cutlass_fused_moe_module(device_arch).cutlass_fused_moe(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/flashinfer/fused_moe/core.py", line 607, in cutlass_fused_moe
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     run_moe(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "python/tvm_ffi/cython/function.pxi", line 923, in tvm_ffi.core.Function.__call__
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "<unknown>", line 0, in FusedMoeRunner::GetFunction(tvm::ffi::String const&)::{lambda(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::Array<tvm::ffi::Tensor, void>, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, long, long, long, long, long, long, bool, bool, tvm::ffi::Optional<tvm::ffi::Array<long, void>, void>, bool, long)#5}::operator()(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::Array<tvm::ffi::Tensor, void>, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, long, long, long, long, long, long, bool, bool, tvm::ffi::Optional<tvm::ffi::Array<long, void>, void>, bool, long) const
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "<unknown>", line 0, in FusedMoeRunner::runMoe(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::Array<tvm::ffi::Tensor, void>, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, long, long, long, long, long, long, bool, bool, tvm::ffi::Optional<tvm::ffi::Array<long, void>, void>, bool, tensorrt_llm::kernels::cutlass_kernels::ActivationType) [clone .isra.0]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "<unknown>", line 0, in FusedMoeRunner::getWorkspaceInfo(long, long, long, int, int, tensorrt_llm::kernels::cutlass_kernels::ActivationType, tensorrt_llm::kernels::cutlass_kernels::MOEParallelismConfig, bool)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "<unknown>", line 0, in TVMFFIEnvTensorAlloc
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "<unknown>", line 0, in TorchDLPackExchangeAPI::ManagedTensorAllocator(DLTensor*, DLManagedTensorVersioned**, void*, void (*)(void*, char const*, char const*))
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] MemoryError: CUDA out of memory. Tried to allocate 2.27 GiB. GPU 4 has a total capacity of 79.19 GiB of which 1.43 GiB is free. Including non-PyTorch memory, this process has 77.75 GiB memory in use. Of the allocated memory 72.10 GiB is allocated by PyTorch, with 536.00 MiB allocated in private pools (e.g., CUDA Graphs), and 1.81 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] Exception raised from malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:1574 (most recent call first):
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x7fb31b0ddfdd in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #1: <unknown function> + 0x42555 (0x7fb31b1ad555 in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #2: <unknown function> + 0x427e2 (0x7fb31b1ad7e2 in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #3: <unknown function> + 0x42e9f (0x7fb31b1ade9f in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #4: at::detail::empty_generic(c10::ArrayRef<long>, c10::Allocator*, c10::DispatchKeySet, c10::ScalarType, std::optional<c10::MemoryFormat>) + 0x5cc (0x7fb231797a7c in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #5: at::detail::empty_cuda(c10::ArrayRef<long>, c10::ScalarType, std::optional<c10::Device>, std::optional<c10::MemoryFormat>) + 0x7f (0x7fb1d7ddbcbf in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #6: at::detail::empty_cuda(c10::ArrayRef<long>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>) + 0x6e (0x7fb1d7ddbece in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #7: at::native::empty_cuda(c10::ArrayRef<long>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>) + 0x43 (0x7fb1d80d1a33 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #8: <unknown function> + 0x37d9c52 (0x7fb1da64cc52 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #9: <unknown function> + 0x37d9d6c (0x7fb1da64cd6c in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #10: at::_ops::empty_memory_format::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>) + 0xe6 (0x7fb2328c7e36 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #11: <unknown function> + 0x2ee5b19 (0x7fb232d2fb19 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #12: at::_ops::empty_memory_format::call(c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>) + 0x154 (0x7fb23299c974 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #13: <unknown function> + 0x76ba05 (0x7fb245a1fa05 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #14: TVMFFIEnvTensorAlloc + 0x68 (0x7fadd47b2a28 in /opt/venv/lib/python3.12/site-packages/tvm_ffi/lib/libtvm_ffi.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #15: FusedMoeRunner::getWorkspaceInfo(long, long, long, int, int, tensorrt_llm::kernels::cutlass_kernels::ActivationType, tensorrt_llm::kernels::cutlass_kernels::MOEParallelismConfig, bool) + 0x17e (0x7fa5cd538dae in /opt/venv/lib/python3.12/site-packages/flashinfer_jit_cache/jit_cache/fused_moe_90/fused_moe_90.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #16: <unknown function> + 0x78e302 (0x7fa5cd560302 in /opt/venv/lib/python3.12/site-packages/flashinfer_jit_cache/jit_cache/fused_moe_90/fused_moe_90.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #17: FusedMoeRunner::GetFunction(tvm::ffi::String const&)::{lambda(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::Array<tvm::ffi::Tensor, void>, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, long, long, long, long, long, long, bool, bool, tvm::ffi::Optional<tvm::ffi::Array<long, void>, void>, bool, long)#5}::operator()(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::Array<tvm::ffi::Tensor, void>, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, long, long, long, long, long, long, bool, bool, tvm::ffi::Optional<tvm::ffi::Array<long, void>, void>, bool, long) const + 0x2de (0x7fa5cd566dbe in /opt/venv/lib/python3.12/site-packages/flashinfer_jit_cache/jit_cache/fused_moe_90/fused_moe_90.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #18: tvm::ffi::Function::FromTyped<FusedMoeRunner::GetFunction(tvm::ffi::String const&)::{lambda(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::Array<tvm::ffi::Tensor, void>, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, long, long, long, long, long, long, bool, bool, tvm::ffi::Optional<tvm::ffi::Array<long, void>, void>, bool, long)#5}>(FusedMoeRunner::GetFunction(tvm::ffi::String const&)::{lambda(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::Array<tvm::ffi::Tensor, void>, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, long, long, long, long, long, long, bool, bool, tvm::ffi::Optional<tvm::ffi::Array<long, void>, void>, bool, long)#5}&&)::{lambda(tvm::ffi::AnyView const*, int, tvm::ffi::Any*)#1}::operator()(tvm::ffi::AnyView const*, int, tvm::ffi::Any*) + 0x12657 (0x7fa5cd58a0d7 in /opt/venv/lib/python3.12/site-packages/flashinfer_jit_cache/jit_cache/fused_moe_90/fused_moe_90.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #19: tvm::ffi::details::FunctionObjImpl<tvm::ffi::Function::FromTyped<FusedMoeRunner::GetFunction(tvm::ffi::String const&)::{lambda(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::Array<tvm::ffi::Tensor, void>, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, long, long, long, long, long, long, bool, bool, tvm::ffi::Optional<tvm::ffi::Array<long, void>, void>, bool, long)#5}>(FusedMoeRunner::GetFunction(tvm::ffi::String const&)::{lambda(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::TensorView, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::Array<tvm::ffi::Tensor, void>, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, tvm::ffi::Optional<tvm::ffi::TensorView, void>, long, long, long, long, long, long, bool, bool, tvm::ffi::Optional<tvm::ffi::Array<long, void>, void>, bool, long)#5}&&)::{lambda(tvm::ffi::AnyView const*, int, tvm::ffi::Any*)#1}>::SafeCall(void*, TVMFFIAny const*, int, TVMFFIAny*) + 0x373 (0x7fa5cd53f3b3 in /opt/venv/lib/python3.12/site-packages/flashinfer_jit_cache/jit_cache/fused_moe_90/fused_moe_90.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #20: <unknown function> + 0x54a8c (0x7fb0c4063a8c in /opt/venv/lib/python3.12/site-packages/tvm_ffi/core.abi3.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #21: PyObject_Call + 0x6c (0x54afac in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #22: _PyEval_EvalFrameDefault + 0x4b7a (0x5db2ca in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #23: VLLM::Worker_TP4_EP4() [0x54ca34]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #24: PyObject_Call + 0x115 (0x54b055 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #25: _PyEval_EvalFrameDefault + 0x4b7a (0x5db2ca in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #26: VLLM::Worker_TP4_EP4() [0x54ca34]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #27: PyObject_Call + 0x115 (0x54b055 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #28: _PyEval_EvalFrameDefault + 0x4b7a (0x5db2ca in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #29: _PyObject_Call_Prepend + 0x18a (0x54a73a in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #30: VLLM::Worker_TP4_EP4() [0x5a3398]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #31: _PyObject_MakeTpCall + 0x13e (0x548eee in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #32: _PyEval_EvalFrameDefault + 0xa89 (0x5d71d9 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #33: <unknown function> + 0xa49f36 (0x7fb245cfdf36 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #34: <unknown function> + 0xdc0d85 (0x7fb246074d85 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #35: <unknown function> + 0x66a755c (0x7fb2364f155c in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #36: <unknown function> + 0xb13864 (0x7fb245dc7864 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #37: <unknown function> + 0xb13d68 (0x7fb245dc7d68 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #38: <unknown function> + 0x9f7c52 (0x7fb245cabc52 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #39: <unknown function> + 0x40e440 (0x7fb2456c2440 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #40: VLLM::Worker_TP4_EP4() [0x581e4f]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #41: PyObject_Call + 0x6c (0x54afac in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #42: _PyEval_EvalFrameDefault + 0x4b7a (0x5db2ca in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #43: _PyObject_Call_Prepend + 0xc2 (0x54a672 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #44: VLLM::Worker_TP4_EP4() [0x5a3398]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #45: _PyObject_MakeTpCall + 0x75 (0x548e25 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #46: _PyEval_EvalFrameDefault + 0xa89 (0x5d71d9 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #47: _PyObject_Call_Prepend + 0xc2 (0x54a672 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #48: VLLM::Worker_TP4_EP4() [0x5a3398]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #49: _PyObject_MakeTpCall + 0x75 (0x548e25 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #50: _PyEval_EvalFrameDefault + 0xa89 (0x5d71d9 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #51: _PyObject_Call_Prepend + 0xc2 (0x54a672 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #52: VLLM::Worker_TP4_EP4() [0x5a3398]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #53: _PyObject_MakeTpCall + 0x75 (0x548e25 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #54: _PyEval_EvalFrameDefault + 0xa89 (0x5d71d9 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #55: _PyObject_Call_Prepend + 0xc2 (0x54a672 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #56: VLLM::Worker_TP4_EP4() [0x5a3398]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #57: PyObject_Call + 0x6c (0x54afac in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #58: _PyEval_EvalFrameDefault + 0x4b7a (0x5db2ca in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #59: _PyObject_Call_Prepend + 0xc2 (0x54a672 in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #60: VLLM::Worker_TP4_EP4() [0x5a3398]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #61: PyObject_Call + 0x6c (0x54afac in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] frame #62: _PyEval_EvalFrameDefault + 0x4b7a (0x5db2ca in VLLM::Worker_TP4_EP4)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863] Traceback (most recent call last):
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 858, in worker_busy_loop
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     output = func(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 361, in execute_model
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.worker.execute_model(scheduler_output)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return func(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 652, in execute_model
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     output = self.model_runner.execute_model(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return func(*args, **kwargs)
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3538, in execute_model
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     model_output = self._model_forward(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]                    ^^^^^^^^^^^^^^^^^^^^
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3047, in _model_forward
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]     return self.model(
(Worker_TP4_EP4 pid=959) ERROR 02-10 19:36:29 [multiproc_executor.py:863]            ^^^^^^^^^^^
...

Root Cause

The error itself just means FLASHINFER_CUTLASS Fp8 MoE backend does not fit well with chunked prefill, but on hopper GPUs we have a preffered Fp8 MoE backend - DeepGEMM. If we force the DeepGEMM backend, the error above is gone.

Fp8 MoE backend selection is done at here, as we can see FLASHINFER_CUTLASS has higher priority than DEEPGEMM, regardless of the GPU arch. We can force-use DeepGEMM backend by setting VLLM_USE_FLASHINFER_MOE_FP8=0 environment variable, up to a caveat that we are not "enabling DeepGEMM" but "disabling flashinfer" - which is counter-intuitive at best.

Suggestion

I feel like flashinfer kernels are mostly for blackwell devices, and vllm already distinguish the kernel selection priority by GPU archs at several places (e.g. flashattn is preffered over flashinfer on hopper and vice versa on blackwell), so it would be natural to apply the same principle and default the Fp8 MoE backend to DEEPGEMM on hopper / FLASHINFER_CUTLASS on blackwell.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[robertgshaw2-redhat]

Thanks for reporting this

[rohan-reddy]

I'd like to try to take this one if that's okay!

[elizabetht]

#34260

[rohan-reddy]

I finished my PR with similar changes shortly after yours, so I'll close it . A couple assumptions to check:

1. For all pre-Hopper GPUs we should also prefer DeepGEMM?
2. Should DeepGEMM be above Flash Infer on a Triton LLM backend, or just above CUTLASS?