[Feature]: Support compressed-tensors NVFP4 quantization for MoE models (Nemotron-H non-gated MoE)

[Firworksyt]

Your current environment

Using nightly-0d4044edd85de30d7d4558aeea4d1e95c7c556d6

• CUDA: 13.0 (also reproduced on 12.8)
• GPU: RTX Pro 6000 Blackwell

🐛 Describe the bug

Compressed-tensors (llm-compressor) NVFP4-quantized MoE models fail to initialize in vLLM due to a hard-coded limitation in the fused MoE layer when is_act_and_mul=False.

This affects Nemotron-H / Nemotron-3-Nano-30B-A3B style MoE models that use a non-gated MoE path.

The failure occurs even though NVFP4 is supported for dense models and even though NVFP4 is explicitly allowed for MoE only when the checkpoint is detected as ModelOpt NVFP4.

Compressed-tensors NVFP4 is currently not recognized as a supported MoE quantization backend.

Reproduction

Model

• Firworks/NVIDIA-Nemotron-3-Nano-30B-A3B-nvfp4
• Quantized using llm-compressor / compressed-tensors
• Quantization recipe:

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=[
        "lm_head",
        "re:visual.*",
        "re:.*vision_tower.*",
        "re:.*video_tower.*",
        "re:.*audio_tower.*",
        "re:.*multi_modal_projector.*",
    ],
)

Command

docker run --runtime nvidia --gpus all \
  -p 8000:8000 --ipc=host \
  vllm/vllm-openai:nightly \
  --model Firworks/NVIDIA-Nemotron-3-Nano-30B-A3B-nvfp4 \
  --dtype auto \
  --max-model-len 32768 \
  --trust-remote-code

---

Issue

Engine initialization fails during MoE layer construction with:

NotImplementedError: is_act_and_mul=False is supported only for
unquantized, ModelOpt FP8, and ModelOpt NvFp4 checkpoints

The error originates from:

vllm/model_executor/layers/fused_moe/layer.py

during initialization of SharedFusedMoE.

---

If you have any suggestions for workarounds or patches I can try I'd be happy to test them out and report back.

Including the full trace here at the end:

(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895] EngineCore failed to start.
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895] Traceback (most recent call last):
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 886, in run_engine_core
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 651, in __init__
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     super().__init__(
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 105, in __init__
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     self._init_executor()
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 48, in _init_executor
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     self.driver_worker.load_model()
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 275, in load_model
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     self.model_runner.load_model(eep_scale_up=eep_scale_up)    
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3781, in load_model
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     self.model = model_loader.load_model(
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]                  ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 49, in load_model
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     model = initialize_model(
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]             ^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/utils.py", line 48, in initialize_model
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     return model_class(vllm_config=vllm_config, prefix=prefix) 
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 823, in __init__
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     self.model = NemotronHModel(
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]                  ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 291, in __init__
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     old_init(self, **kwargs)
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 591, in __init__
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     self.start_layer, self.end_layer, self.layers = make_layers(
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]                                                     ^^^^^^^^^^^^
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 606, in make_layers
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))   
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^    
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 581, in get_layer
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     return layer_class(
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]            ^^^^^^^^^^^^
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 359, in __init__
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     self.mixer = NemotronHMoE(
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]                  ^^^^^^^^^^^^^
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 188, in __init__
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     self.experts = SharedFusedMoE(
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]                    ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/shared_fused_moe.py", line 28, in __init__
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     super().__init__(**kwargs)
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/layer.py", line 605, in __init__
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895]     raise NotImplementedError(
(EngineCore_DP0 pid=257) ERROR 01-05 21:59:37 [core.py:895] NotImplementedError: is_act_and_mul=False is supported only for unquantized , ModelOpt FP8, and ModelOpt NvFp4 checkpoints
(EngineCore_DP0 pid=257) Process EngineCore_DP0:
(EngineCore_DP0 pid=257) Traceback (most recent call last):
(EngineCore_DP0 pid=257)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=257)     self.run()
(EngineCore_DP0 pid=257)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=257)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 899, in run_engine_core
(EngineCore_DP0 pid=257)     raise e
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 886, in run_engine_core
(EngineCore_DP0 pid=257)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=257)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 651, in __init__    
(EngineCore_DP0 pid=257)     super().__init__(
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 105, in __init__    
(EngineCore_DP0 pid=257)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=257)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=257)     self._init_executor()
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 48, in _init_executor
(EngineCore_DP0 pid=257)     self.driver_worker.load_model()
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 275, in load_model
(EngineCore_DP0 pid=257)     self.model_runner.load_model(eep_scale_up=eep_scale_up)
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3781, in load_model
(EngineCore_DP0 pid=257)     self.model = model_loader.load_model(
(EngineCore_DP0 pid=257)                  ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 49, in load_model
(EngineCore_DP0 pid=257)     model = initialize_model(
(EngineCore_DP0 pid=257)             ^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/utils.py", line 48, in initialize_model
(EngineCore_DP0 pid=257)     return model_class(vllm_config=vllm_config, prefix=prefix)
(EngineCore_DP0 pid=257)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 823, in __init__
(EngineCore_DP0 pid=257)     self.model = NemotronHModel(
(EngineCore_DP0 pid=257)                  ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 291, in __init__
(EngineCore_DP0 pid=257)     old_init(self, **kwargs)
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 591, in __init__
(EngineCore_DP0 pid=257)     self.start_layer, self.end_layer, self.layers = make_layers(
(EngineCore_DP0 pid=257)                                                     ^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 606, in make_layers
(EngineCore_DP0 pid=257)     maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
(EngineCore_DP0 pid=257)                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 581, in get_layer
(EngineCore_DP0 pid=257)     return layer_class(
(EngineCore_DP0 pid=257)            ^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 359, in __init__
(EngineCore_DP0 pid=257)     self.mixer = NemotronHMoE(
(EngineCore_DP0 pid=257)                  ^^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 188, in __init__
(EngineCore_DP0 pid=257)     self.experts = SharedFusedMoE(
(EngineCore_DP0 pid=257)                    ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/shared_fused_moe.py", line 28, in __init__
(EngineCore_DP0 pid=257)     super().__init__(**kwargs)
(EngineCore_DP0 pid=257)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/layer.py", line 605, in __init__
(EngineCore_DP0 pid=257)     raise NotImplementedError(
(EngineCore_DP0 pid=257) NotImplementedError: is_act_and_mul=False is supported only for unquantized , ModelOpt FP8, and ModelOpt NvFp4 checkpoints
[rank0]:[W105 21:59:37.864829781 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1)   File "/usr/local/bin/vllm", line 10, in <module>
(APIServer pid=1)     sys.exit(main())
(APIServer pid=1)              ^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=1)     args.dispatch_function(args)
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 60, in cmd
(APIServer pid=1)     uvloop.run(run_server(args))
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=1)     return __asyncio.run(
(APIServer pid=1)            ^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=1)     return runner.run(main)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1)     return self._loop.run_until_complete(task)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1)     return await main
(APIServer pid=1)            ^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1324, in run_server
(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1343, in run_server_worker
(APIServer pid=1)     async with build_async_engine_client(
(APIServer pid=1)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 171, in build_async_engine_client
(APIServer pid=1)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 212, in build_async_engine_client_from_engine_args
(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 207, in from_vllm_config
(APIServer pid=1)     return cls(
(APIServer pid=1)            ^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 134, in __init__      
(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
(APIServer pid=1)     return AsyncMPClient(*client_args)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 824, in __init__    
(APIServer pid=1)     super().__init__(
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 479, in __init__    
(APIServer pid=1)     with launch_core_engines(vllm_config, executor_class, log_stats) as (
(APIServer pid=1)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=1)     next(self.gen)
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 921, in launch_core_engines
(APIServer pid=1)     wait_for_engine_startup(
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 980, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[baonudesifeizhai]

trying to work on it currently non-gated moe seems have lots of problems

[robertgshaw2-redhat]

Thanks @baonudesifeizhai - I actually think these issues are unrelated.

#31957 is due to the MoE refactor. This issue is due to ct not supported non-gated models. Can you take on this fix?

[baonudesifeizhai]

thats the initialize issue
https://paste.ubuntu.com/p/3Cg4HfTHxs/ meet this during the fixing
(EngineCore_DP0 pid=1163609) ERROR 01-08 03:10:07 [core.py:900] RuntimeError: Worker failed with error 'Only SiLU activation is supported, got relu2_no_mul.', please check the stack trace above for the root cause

maybe i went the wrong way...

Thanks @baonudesifeizhai - I actually think these issues are unrelated.

#31957 is due to the MoE refactor. This issue is due to ct not supported non-gated models. Can you take on this fix?

[baonudesifeizhai]

god finally work i will creat a pr