[Bug]: vllm fails to load continual pre-trained Qwen3.5-MoE model due to missing support for transformers 5.x renamed class (Qwen3_5MoeTextConfig)

[itacora]

Your current environment

The output of python collect_env.py

- vllm: 0.16.0 / 0.16.1rc1.dev133+g9a87b0578 / 0.17.0rc0 nightly
- transformers: 5.2.0 / 5.3.0
- Python: 3.12.12
- pytorch: 2.10.0
- cuda: 12.9

🐛 Describe the bug

LLM models

• Base model: Qwen/Qwen3.5-122B-A10B (MoE)
• Model type: CPT (Continued Pre-Training) from base model using transformers 5.3.0

Bug Description

vllm fails to load a CPT model based on Qwen3.5-122B-A10B when using transformers >= 5.x.

transformers 5.x renamed Qwen3_5MoeConfig to Qwen3_5MoeTextConfig, but vllm still references the old class name internally. Since the CPT (Continued Pre-Training) was performed using transformers 5.3.0, the output config.json uses the new class names, making it incompatible with the current vllm.

Error

TypeError: Invalid type of HuggingFace config.
Expected type: <class 'vllm.transformers_utils.configs.qwen3_5_moe.Qwen3_5MoeConfig'>
but found type: <class 'transformers.models.qwen3_5_moe.configuration_qwen3_5_moe.Qwen3_5MoeTextConfig'>

Model config.json (base model output)

{
  "architectures": ["Qwen3_5MoeForConditionalGeneration"],
  "model_type": "qwen3_5_moe",
  "transformers_version": "4.57.0.dev0",
}

Model config.json (CPT model output)

{
  "architectures": ["Qwen3_5MoeForCausalLM"],
  "model_type": "qwen3_5_moe_text",
  "transformers_version": "5.3.0",
}

Steps to Reproduce

1. Perform CPT on Qwen/Qwen3.5-122B-A10B using transformers >= 5.x
2. Attempt to serve the CPT checkpoint with vllm

Expected Behavior

vllm should support transformers 5.x and correctly handle Qwen3_5MoeTextConfig for CPT models based on Qwen3.5-MoE.

#Possible Fix

In vllm/transformers_utils/configs/qwen3_5_moe.py and related files, update the class reference from Qwen3_5MoeConfig to Qwen3_5MoeTextConfig to match transformers 5.x naming convention.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[MatteoFari]

vLLM already has Qwen3_5MoeTextConfig. The issue is that text-only Qwen3.5-MoE checkpoints saved by newer Transformers are being routed through the multimodal Qwen3.5-MoE path, so vLLM expects Qwen3_5MoeConfig instead of the text-only config and fails the type check.

The fix should be to add proper support for the text-only Qwen3.5-MoE model type and ForCausalLM architecture, rather than replacing the multimodal config class everywhere. What do you think?

[itacora]

Thanks for the clarification, that makes sense.

If the issue is the routing of "qwen3_5_moe_text" checkpoints through the multimodal path, then adding proper support for the text-only Qwen3.5-MoE model type sounds like the right approach.

[ywang96]

To clarify - we're totally okay with registering a text-only Qwen3.5-MoE model type inside vLLM. The only issue with that is we need a public huggingface model repository with its architecture actually registered as Qwen3_5MoeForCausalLM so that we can test model loading on our CI with it (This is a requirement for every new model registration).

[ferreroal]

hi @ywang96 i am having the same problem. I tried launching with the option --language-model-only as you mention in #36289 but it didn't worked

[djaffer]

| (APIServer pid=1) TypeError: Invalid type of HuggingFace config. Expected type: <class 'vllm.transformers_utils.configs.qwen3_5.Qwen3_5Config'>, but found type: <class 'transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig'> same issue tried using qwen-3.5-4b model gptq quantized.

[PathosEthosLogos]

Unsure if unrelated, but I have been using Docker compose, so nothing was changed and I had to restart the container due to maintenance (no system updates or anything like that either, no Docker updates, nothing), but just got this error: AttributeError: 'Qwen3_5MoeTextConfig' object has no attribute 'set'

I was guessing (probably a bad guess) it's because --trust-remote-code ran something to update inside the image/container or something along the lines, but no way to check.

What could possibly explain this?

[djaffer]

I was not able to run. https://huggingface.co/RafaDom/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2-GPTQ-Int4-HQ Created a new gptq model as well same error. I want image too not just text.

[Oxi84]

vLLM already has Qwen3_5MoeTextConfig. The issue is that text-only Qwen3.5-MoE checkpoints saved by newer Transformers are being routed through the multimodal Qwen3.5-MoE path, so vLLM expects Qwen3_5MoeConfig instead of the text-only config and fails the type check.

The fix should be to add proper support for the text-only Qwen3.5-MoE model type and ForCausalLM architecture, rather than replacing the multimodal config class everywhere. What do you think?

Also replacing multimodal class make the model work but now aways mostly there are errors and garbage produced as well