[Feature]: MoE LoRA for quantized models

[jeejeelee]

🚀 The feature, motivation and pitch

This issue tracks and records the LoRA support status for different quantization methods in vLLM.

Current Support Status

Quantization Method	LoRA Support	Test Model	Notes
MXFP4	✅	gpt-oss-20b
FP8	✅	qwen3-30b-a3b-fp8
GPTQ	✅	qwen3-30b-a3b-gptq-int4
AWQ	✅	qwen3-30b-a3b-awq
compressed_tensors	✅	qwen3-30b-a3b-w4a16
Bitsandbytes	❌	qwen3-30b-a3b-bnb	Offline Quantization
Bitsandbytes	❌	qwen3-30b-a3b	Online Quantization
GGUF	❌	qwen3-30b-a3b-gguf

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[classicsong]

Some notes: Right now gpt-oss-20b only works with Marlin kernel, support for triton kernel is still [In progress] (see #28971 )
Qwen-30b-a3b-fp8 only works with triton fp8 kernel. It does not work with CUTLASS kernel.

[Bhanu068]

@jeejeelee, is this open for external collaborations? If so, I would like to contribute to some of the action items.

[princepride]

@jeejeelee I also want implement some quantization method.🤭

[jeejeelee]

@Bhanu068 @princepride thank you, contributions are greatly appreciated

[Bhanu068]

@jeejeelee @princepride, I would like to start with GPTQ. @jeejeelee, do you have anything to share that would give me a kickstart? I am exploring the current supported implementations to understand how I should design and implement the new feature.

[princepride]

@jeejeelee I'd like to take on the AWQ implementation. Similar to @Bhanu068, I would also appreciate a kickstart or any pointers on where to look in the codebase to align with the current design pattern.

[jeejeelee]

1. Use the latest main branch
2. Understand how MXP4 and FP16 MoE models support LoRA
3. the key is https://github.com/vllm-project/vllm/blob/main/vllm/lora/layers/fused_moe.py#L127, where the quant_method needs to implement the select_gemm_impl method

[Bhanu068]

@jeejeelee I have added a gptq moe config gptq_marlin_moe_quant_config in vllm/model_executor/layers/fused_moe/config.py and implemented the select_gemm_impl method in vllm/model_executor/layers/quantization/gptq_marlin.py. The problem is that the existing _fused_marlin_moe function calls the moe_wna16_marlin_gemm in csrc/moe/marlin_moe_wna16/ops.cu. This is written based on MXFP4 with a tile_size of 16, but GPTQ requires a tile_size of 8. This raises an assertion error

// Verify B
  TORCH_CHECK(
      size_k % tile_size == 0, "size_k = ", size_k,
      " is not divisible by tile_size = ", tile_size);
  TORCH_CHECK((size_k / tile_size) == b_q_weight.size(1),
              "Shape mismatch: b_q_weight.size(1) = ", b_q_weight.size(1),
              ", size_k = ", size_k,
              ", tile_size = ", tile_size);
  TORCH_CHECK(
      b_q_weight.size(2) % tile_size == 0,
      "b_q_weight.size(2) = ", b_q_weight.size(2),
      " is not divisible by tile_size = ", tile_size);

One of the options is to extract the tile_size - int tile_size = (b_type == vllm::kU4B8) ? 8 : 16; and pass it as a parameter downstream till the final kernel computation. This will allow us to reuse existing kernels that can support both MXFP4 and GPTQ.

I want to get your thoughts on this and if I am on the right track.

[jeejeelee]

@Bhanu068 sorry for missing your message, maybe you can submit a PR first

[Bhanu068]

@jeejeelee Submitted the PR. Ignore my previous comment on tile_size. I realised it is not required.

[princepride]

@jeejeelee Submitted the PR #30442

[Allyyi]

Hi, @jeejeelee I'd like to work on Bitsandbytes for qwen3-30b-a3b-bnb and qwen3-30b-a3b. I'll try to get this done within a week. :)