[New Model]: Qwen 3.5 support from huggingface's transformers v5.2+

[ernestyalumni]

The model to consider.

Release 1.3.0rc8 in the NGC (i.e. NVIDIA's dockers repo) catalog
https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags?version=1.3.0rc8
uses
tensorrt_llm 1.3.0rc8
transformers 4.57.3
(just run pip list after running (docker exec ...) to "docker exec" into the docker container once it's running.
You are not able to deploy Qwen3.5 models with this version of transformers, 4.X.

Qwen3.5 was supported from v5.2 onwards:
https://github.com/huggingface/transformers/releases/tag/v5.2.0

If you update transformers to v5.X + you immediately run into this error when trying to run trtllm-serve:

  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/gpt/model.py", line 32, in <module>
    from .convert import (load_hf_gpt, load_weights_from_hf_model,
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/gpt/convert.py", line 32, in <module>
    from transformers import (AutoModelForCausalLM, AutoModelForVision2Seq,
ImportError: cannot import name 'AutoModelForVision2Seq' from 'transformers' (/ThirdParty/transformers/src/transformers/__init__.py)

This is still within the NGC docker.

AutoModelForVision2Seq was hard deprecated by any v5 of transformers:
https://github.com/huggingface/transformers/blob/779cd2d6923c9d264d46136e90c58f8bafade6ab/MIGRATION_GUIDE_V5.md?plain=1#L506

I see that the code on the main branch is still referring to this deprecated class on this commit snapshot b095c62 : and in these files:

TensorRT-LLM/tensorrt_llm/models/gpt/convert.py

Line 32 in 34082e4

from transformers import (AutoModelForCausalLM, AutoModelForVision2Seq,

TensorRT-LLM/tensorrt_llm/tools/multimodal_builder.py

Line 18 in 34082e4

AutoModelForVision2Seq, AutoProcessor,

Can we

• Update to using AutoModelForImageTextToText and remove any AutoModelForVision2Seq; we need to do that anyways
• Check if any existing models that need to be critical be supported break from this change (so far I only see a kosmos)
• Then check if a Docker container with the updated tensorRT-llm and updated transformers works when running

• trtllm-serve
• trtllm-eval
• trtllm-bench

• Finally be able to run Qwen3.5 models with trtllm-serve with this Docker container

The closest model TensorRT-LLM already supports.

No response

What's your difficulty of supporting the model you want?

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[karljang]

@ernestyalumni ,
Thank you for opening this issue!
The Transformers version has not been updated yet. However, I wanted to inform you that there was a PR that added support for Qwen3.5 text models first~ :)

[CDalezyb]

@ernestyalumni , Thank you for opening this issue! The Transformers version has not been updated yet. However, I wanted to inform you that there was a PR that added support for Qwen3.5 text models first~ :)

so, since PR 12242 added supports for Qwen3.5 and release v1.3.0rc9 announced support for Qwen3.5, how can we run Qwen3.5 in Pytorch backend through the release image? The default transformers is 4.57.3 which doesn't support Qwen3_5 model and after i upgrade transformers, i got problem imports deprecated Transformers (see issues 13822

[karljang]

#13791 upgrades transformers version to 5.3.0, in progress~

[karljang]

Closing this as the comments above should have addressed it. Please open a new one if it persists on the latest version