[Model] Use merge_by_field_config for MM models (M-N) #26710
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337
added
the
ready
mergify
bot
added
llama
13 tasks
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request refactors multi-modal data handling across several models to use a new, centralized framework enabled by merge_by_field_config = True. This is a significant improvement that simplifies model-specific code by removing manual data validation, batching, and token ID handling, leading to cleaner and more maintainable code. The changes are mostly consistent. My review focuses on a couple of areas in nano_nemotron_vl.py where the refactoring appears incomplete, resulting in unused code that should be removed to fully align with the PR's goals.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Isotr0py
approved these changes
Oct 13, 2025
1994
pushed a commit
to 1994/vllm
that referenced
this pull request
Oct 14, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: 1994 <1994@users.noreply.github.com>
Dhruvilbhatt
pushed a commit
to Dhruvilbhatt/vllm
that referenced
this pull request
Oct 14, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Dhruvil Bhatt <bhattdbh@amazon.com>
bbartels
pushed a commit
to bbartels/vllm
that referenced
this pull request
Oct 16, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: bbartels <benjamin@bartels.dev>
lywa1998
pushed a commit
to lywa1998/vllm
that referenced
this pull request
Oct 20, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
alhridoy
pushed a commit
to alhridoy/vllm
that referenced
this pull request
Oct 24, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
xuebwang-amd
pushed a commit
to xuebwang-amd/vllm
that referenced
this pull request
Oct 24, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
xuebwang-amd
pushed a commit
to xuebwang-amd/vllm
that referenced
this pull request
Oct 24, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
0xrushi
pushed a commit
to 0xrushi/vllm
that referenced
this pull request
Oct 26, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
0xrushi
pushed a commit
to 0xrushi/vllm
that referenced
this pull request
Oct 26, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
rtourgeman
pushed a commit
to rtourgeman/vllm
that referenced
this pull request
Nov 10, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Zhathw
pushed a commit
to Zhathw/vllm
that referenced
this pull request
Nov 12, 2025
…6710) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Part of #26149
Also remove unnecessary token IDs from the field config, since we have removed V0 specific code in #25331 already
Test Plan
Tensor schema tests should pass.
I have also run
MiDashengLMModel,MiniCPMO,MiniCPMVandMolmoForCausalLMusing the example scripts and confirmed they still work.Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.