[Bug]: qwen 3.5 9B quantized lm_head is not supported in vLLM (autoround format)

[xin3he]

Problem Description

Per @wenhuach21's comments, we had supported quantized lm_head in vLLM but now it no longer work.
We'd better find out the reason and fix it.

Reproduction Steps

1. auto-round /workspace/models/Qwen/Qwen3.5-9B/ --quant_lm_head --iters 0 --disable_opt_rtn
2. vllm serve tmp_autoround/Qwen3.5-9B-w4g128/ --port 8000 --tensor-parallel-size 1 --max-model-len 2048 --reasoning-parser qwen3 --served-model-name qwen --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

Environment Information

No response

Error Logs

Additional Context

No response

[xin3he]

For https://huggingface.co/OPEA/glm-4-9b-chat-int4-sym-lm-head-inc, quantized lm_head works well.
vllm serve /workspace/models/OPEA/glm-4-9b- chat-int4-sym-lm-head-inc/ --port 8000 --tensor-parallel-size 1 --max-model-len 2048 --se rved-model-name qwen --trust_remote_code

[vishvaRam]

I'm not sure if this is directly related to this issue or not, but I wanted to share my findings in case they are helpful.

I recently requantized the Qwen 3.5 9B model using AutoRound, and it is working perfectly in vLLM, including support for turboquant with the KV cache.

For context, the requantization reduced the storage footprint significantly, dropping it from 17.8 GB down to 9.08 GB. The uploaded model is available here: Vishva007/Qwen3.5-9B-W4A16-AutoRound.

You can find the exact code and configuration I used for the requantization in this notebook reference: auto_round_Qwen3_5_9B.ipynb.

I tested the resulting model successfully using the vLLM nightly Docker image (vllm/vllm-openai:nightly-28ee78af543c563a2fbf78829a7688120e4e4eb5).

Here is the launch command I used to serve it:

vllm serve Vishva007/Qwen3.5-9B-W4A16-AutoRound \
  --port 8000 \
  --host 0.0.0.0 \
  --max-model-len 18000 \
  --kv-cache-dtype turboquant_4bit_nc \
  --gpu-memory-utilization 0.80 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 18000 \
  --max-num-seqs 64 \
  --trust-remote-code \
  --quantization inc \
  --dtype bfloat16 \
  --mm-processor-kwargs '{"max_pixels": 10485760, "max_length": 65536, "truncation": false}' \
  --mm-processor-cache-type shm \
  --limit-mm-per-prompt.image 50 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking": false}'

Also, in my testing, speculative decoding with:

--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

also works well. However, mtp could not be used together with turboquant, so in that case it is better to set --kv-cache-dtype to int8_per_token_head or fp8, depending on what your hardware supports.

[wenhuach21]

May I know whether we have identified the root cause yet, @xin3he?

[xin3he]

May I know whether we have identified the root cause yet, @xin3he?

fixed in vllm side, thanks.