Qwen3.5 Model Tokenizer Loading Failure

[ArtemSultanov-PG]

Environment

• vLLM version: 0.16.0rc2.dev376+gf4af642a6 (qwen3_5-cu130) and 0.16.1rc1.dev48+ga572baff5 (nightly)
• Docker image: vllm/vllm-openai:qwen3_5-cu130 and vllm/vllm-openai:nightly
• Host environment: Linux, CUDA 13.0
• Python version: 3.12

Reproduction Steps

1. Pull the Qwen3.5-4B model from HuggingFace:

huggingface-cli download Qwen/Qwen3.5-4B --local-dir /data/models/Qwen3.5-4B

2. Try to run with vLLM:

docker run --gpus device=7 \
  -v /data/models:/data/models \
  -p 8000:8000 \
  -e HF_TOKEN=xxx \
  vllm/vllm-openai:qwen3_5-cu130 \
  --model /data/models/Qwen3.5-4B \
  --dtype bfloat16 \
  --host 0.0.0.0 \
  --port 8000

Expected Behavior
Model should load and start the API server.

Actual Error

RuntimeError: Failed to load the tokenizer. If the tokenizer is a custom tokenizer not yet available in the HuggingFace transformers library, consider setting `trust_remote_code=True` in LLM or using the `--trust-remote-code` flag in the CLI.

Full stack trace:

ValueError: Tokenizer class TokenizersBackend does not exist or is not currently imported.

Model Details

• Model: Qwen/Qwen3.5-4B (HuggingFace)
• Config: model_type: qwen3_5
• Transformers version in model: 5.3.0.dev0

Investigation
The model config shows it was built with transformers >= 5.0, but vLLM currently depends on transformers < 5.0:

• vLLM qwen3_5-cu130: requires transformers>=4.56.0,<5.0.0

Suggested Fix

1. Option A: Update vLLM dependencies to support transformers>=5.0 for Qwen3.5 models
2. Option B: Add proper tokenizer support for qwen3_5 in the current transformers version range
3. Option C: Bundle a compatible tokenizer with the vLLM image

Additional Notes

• The --trust-remote-code flag is ignored in newer vLLM versions
• Other Qwen3 models (non-3.5) work fine with the same vLLM versions
• This affects all Qwen3.5 variants (4B, 8B, etc.)

[ArtemSultanov-PG]

Root Cause Analysis

Problem: Both Docker vLLM nightly (0.16.1rc1) and Native Python vLLM (0.16.0) fail to load Qwen3.5-4B (Dense model).

Error (identical on both platforms):

pydantic_core._pydantic_core.ValidationError: 
  Value error, Model architectures [Qwen3_5ForConditionalGeneration] are not supported for now.

Key Findings:

1. Two different Qwen3.5 architectures:

   • Qwen3_5MoeForConditionalGeneration (MoE) — supported in vLLM nightly (Cyanwiki adaptation)
   • Qwen3_5ForConditionalGeneration (Dense) — NOT supported in any vLLM version tested

2. Problem is NOT version-specific:

   • Both Docker nightly AND native Python 0.16.0 fail with same error
   • Issue is in vLLM core library, not Docker image or transformers version

3. Working model: Cyanwiki's Qwen3-Coder-Next-AWQ-4bit uses Qwen3_5MoeForConditionalGeneration — specifically adapted for vLLM.

Conclusion: vLLM core library lacks support for Qwen3_5ForConditionalGeneration architecture from Qwen/Alibaba. This is a feature request for vLLM, not a configuration issue.

[thbupt]

same question

[Heiheicxx]

来信已收到^-^谢谢你的来信，保持联系

[MetaPeak]

pip install -U transformers to update the transorformers to the latest version, for now, it is 5.4.0, solves my problems.

[Heiheicxx]

来信已收到^-^谢谢你的来信，保持联系