[None][fix] Fix Qwen2-VL Transformers 5 compatibility#15997
Conversation
* Why? Transformers 5.x returns vision features in a pooling wrapper, and MRoPE position tensors must be on the model device for inference. * What? Extract pooled vision embeddings, transfer MRoPE tensors with multimodal data, and add regression coverage for both paths. Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
📝 WalkthroughWalkthroughModifies Qwen2VL vision model forward pass to extract pooler_output when vision encoder returns BaseModelOutputWithPooling objects, and extends multimodal_data_device_paths to include mrope_config position tensor paths for disaggregated multimodal execution. Adds two corresponding unit tests. ChangesQwen2VL vision pooling and mrope device paths
Estimated code review effort: 2 (Simple) | ~15 minutes Suggested reviewers: Poem: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tensorrt_llm/_torch/models/modeling_qwen2vl.py (1)
1081-1093: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicate pooler-extraction logic across image/video branches.
The
isinstance(embed, BaseModelOutputWithPooling)check and.pooler_outputextraction are repeated verbatim for both image and video paths. Extracting a small helper avoids the two copies silently diverging if this logic needs a future tweak (e.g. additional output types).♻️ Proposed refactor
+ `@staticmethod` + def _extract_vision_embed( + output: Union[torch.Tensor, BaseModelOutputWithPooling] + ) -> torch.Tensor: + if isinstance(output, BaseModelOutputWithPooling): + return output.pooler_output + return output + embeds = [] if pixel_values is not None: - embed = self.visual(pixel_values, grid_thw=image_grid_thw) - if isinstance(embed, BaseModelOutputWithPooling): - embed = embed.pooler_output - embeds.append(embed) + embeds.append( + self._extract_vision_embed( + self.visual(pixel_values, grid_thw=image_grid_thw))) if pixel_values_videos is not None: - embed = self.visual(pixel_values_videos, grid_thw=video_grid_thw) - if isinstance(embed, BaseModelOutputWithPooling): - embed = embed.pooler_output - embeds.append(embed) + embeds.append( + self._extract_vision_embed( + self.visual(pixel_values_videos, grid_thw=video_grid_thw)))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/models/modeling_qwen2vl.py` around lines 1081 - 1093, The image and video branches in the visual embedding flow duplicate the same BaseModelOutputWithPooling handling logic. Refactor the repeated isinstance(embed, BaseModelOutputWithPooling) and pooler_output extraction into a small helper used by the visual embedding method, so both pixel_values and pixel_values_videos paths share one implementation and stay consistent if the output handling changes later.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tensorrt_llm/_torch/models/modeling_qwen2vl.py`:
- Around line 1081-1093: The image and video branches in the visual embedding
flow duplicate the same BaseModelOutputWithPooling handling logic. Refactor the
repeated isinstance(embed, BaseModelOutputWithPooling) and pooler_output
extraction into a small helper used by the visual embedding method, so both
pixel_values and pixel_values_videos paths share one implementation and stay
consistent if the output handling changes later.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 003f477f-eb63-4dde-a265-64eb7f7b69a5
📒 Files selected for processing (2)
tensorrt_llm/_torch/models/modeling_qwen2vl.pytests/unittest/_torch/modeling/test_modeling_qwen2_5vl.py
|
/bot run |
|
PR_Github #57935 [ run ] triggered by Bot. Commit: |
|
PR_Github #57935 [ run ] completed with state
|
|
/bot run |
|
PR_Github #58058 [ run ] triggered by Bot. Commit: |
|
PR_Github #58058 [ run ] completed with state
|
|
/bot run |
|
PR_Github #58075 [ run ] triggered by Bot. Commit: |
|
PR_Github #58075 [ run ] completed with state |
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
3 participants
Summary by CodeRabbit
Bug Fixes
Tests
Description
Transformers 5.x returns vision features in a pooling wrapper, and MRoPE position tensors must be on the model device for inference.
Extract pooled vision embeddings, transfer MRoPE tensors with multimodal data, and add regression coverage for both paths.
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.