[Usage] Qwen3 Usage Guide

[simon-mo]

vLLM v0.8.4 and higher natively supports all Qwen3 and Qwen3MoE models. Example command:

• vllm serve Qwen/... --enable-reasoning --reasoning-parser deepseek_r1

  • All models should work with the command as above. You can test the reasoning parser with the following example script: https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py
  • Some MoE models might not be divisible by TP 8. Either lower your TP size or use --enable-expert-parallel.

• If you are seeing the following error when running fp8 dense models, you are running on vLLM v0.8.4. Please upgrade to v0.8.5.

File ".../vllm/model_executor/parameter.py", line 149, in load_qkv_weight
    param_data = param_data.narrow(self.output_dim, shard_offset,
IndexError: start out of range (expected to be in range of [-18, 18], but got 2048)

• If you are seeing the following error when running MoE models with fp8, you are running with too much tensor parallelize degree that the weights are not divisible. Consider --tensor-parallel-size 4 or --tensor-parallel-size 8 --enable-expert-parallel.

File ".../vllm/vllm/model_executor/layers/quantization/fp8.py", line 477, in create_weights
    raise ValueError(
ValueError: The output_size of gate's and up's weight = 192 is not divisible by weight quantization block_n = 128.

[sp1cae]

how to use MCP with Qwen3

[NaiveYan]

Any plan for speculative decoding?

[dylanwuzc]

vllm的启动参数如何支持enable_thinking=True？

[DarkLight1337]

vllm的启动参数如何支持enable_thinking=True？

See https://qwen.readthedocs.io/en/latest/deployment/vllm.html#thinking-non-thinking-modes

[thiner]

Consider --tensor-parallel-size 4 or --tensor-parallel-size 8 --enable-expert-parallel.

I am running Qwen3-30B-A3B-FP8 with two A10 GPUs. tp=2 is enough to load the model, does vllm support "tp=2" in this case?

[cmpunk-bitw]

How can I disable reasoning in generative models, i.e. using LLM.chat?

[LiuzRush]

so if i upgrade my vllm version from 0.8.4 to 0.8.5，i don't need to make this fix?

[cyjin-yl]

so if i upgrade my vllm version from 0.8.4 to 0.8.5，i don't need to make this fix?

Refer to the release notes of 0.8.5 (top line), Yes.

Day 0 support for Qwen3 and Qwen3MoE. This release fixes fp8 weight loading (#17318) and adds tuned MoE configs (#17328).

linear.py fixes already existed in #17318

[LiuzRush]

I often fail to follow up on relevant PRs in a timely manner. Thanks for your answer.

so if i upgrade my vllm version from 0.8.4 to 0.8.5，i don't need to make this fix?

Refer to the release notes of 0.8.5 (top line), Yes.

Day 0 support for Qwen3 and Qwen3MoE. This release fixes fp8 weight loading (#17318) and adds tuned MoE configs (#17328).

linear.py fixes already existed in #17318

I often fail to follow up on relevant PRs in a timely manner. Thanks for your answer.

[Vincentdu-cn]

vllm的启动参数如何支持enable_thinking=True？

See https://qwen.readthedocs.io/en/latest/deployment/vllm.html#thinking-non-thinking-modes

The official documentation only describes how to turn off Thinking mode when the API is called, it doesn't write how to turn off Thinking mode as soon as vLLM is started, I tried changing the generation_config.json file to turn off Thinking, but it didn't work, it's still in Thinking mode. I also tried adding “chat_template_kwargs”: {“enable_thinking”: false} via the --override-generation-config parameter, but I don't know the correct usage of this parameter and it keeps giving me errors. Here is my generation_config.json and docker startup command:
generation_config.json:

{
    "bos_token_id": 151643,
    "do_sample": true,
    "eos_token_id": [
        151645,
        151643
    ],
    "pad_token_id": 151643,
    "temperature": 0.7,
    "top_p": 0.8,
    "top_k": 20,
    "max_tokens": 8192,
    "presence_penalty": 1.5,
    "chat_template_kwargs": {"enable_thinking": false},
    "transformers_version": "4.51.0"
}

docker run

 docker run -d --name Qwen --runtime nvidia  --gpus '"device=0,1,2,3"' \
      -v /home/models/Qwen3-32B:/root/.cache/modelscope/hub/Qwen3-32B \
      -p 18081:8081 \
      --ipc=host  \
      vllm/vllm-openai:v0.8.5 \
      --model /root/.cache/modelscope/hub/Qwen3-32B \
      --served-model-name Qwen3-32B \
      --enable-auto-tool-choice --tool-call-parser hermes \
      --chat-template examples/tool_chat_template_hermes.jinja \
      --gpu-memory-utilization 0.9 \
      --tensor-parallel-size 4 \
      --port 8081

ERROR:
--override-generation-config "{'temperature': 0.7,'top_p': 0.8,'top_k': 20,'max_tokens': 8192,'presence_penalty': 1.5,'chat_template_kwargs': {'enable_thinking': false}}"

api_server.py: error: argument --override-generation-config: invalid loads value: "{'temperature': 0.7,'top_p': 0.8,'top_k': 20,'max_tokens': 8192,'presence_penalty': 1.5,'chat_template_kwargs': {'enable_thinking': false}}"

[DarkLight1337]

How can I disable reasoning in generative models, i.e. using LLM.chat?

I have opened #17356 to support this, can you try it?