Move VLM conversions to the main mapping#44627
Move VLM conversions to the main mapping#44627Cyrilvallez merged 20 commits intohuggingface:mainfrom
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
src/transformers/models/ernie4_5_vl_moe/modeling_ernie4_5_vl_moe.py
Outdated
Show resolved
Hide resolved
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
run-slow: llava, qwen2_vl, paligemma, gemma3 |
|
This comment contains models: ["models/gemma3", "models/llava", "models/paligemma", "models/qwen2_vl"] |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
src/transformers/models/sam3_tracker_video/modeling_sam3_tracker_video.py
Show resolved
Hide resolved
|
run-slow: colmodernvbert got_ocr2 paligemma qwen3_5_moe llava_next_video fuyu qwen2_5_vl cohere2_vision lfm2_vl llava_next ovis2 mllama shieldgemma2 qwen3_vl_moe glm4v_moe glm46v aria gemma3 llava llava_onevision mistral3 sam3_tracker fast_vlm florence2 colpali glm4v emu3 aya_vision qwen3_vl vipllava paddleocr_vl lighton_ocr video_llava glm_ocr colqwen2 ernie4_5_vl_moe video_llama_3 glm_image qwen3_5 internvl sam3 gemma3n sam3_tracker_video qwen2_vl perception_lm |
|
This comment contains models: ["models/aria", "models/aya_vision", "models/cohere2_vision", "models/colmodernvbert", "models/colpali", "models/colqwen2", "models/emu3", "models/ernie4_5_vl_moe", "models/fast_vlm", "models/florence2", "models/fuyu", "models/gemma3", "models/gemma3n", "models/glm46v", "models/glm4v", "models/glm4v_moe", "models/glm_image", "models/glm_ocr", "models/got_ocr2", "models/internvl", "models/lfm2_vl", "models/lighton_ocr", "models/llava", "models/llava_next", "models/llava_next_video", "models/llava_onevision", "models/mistral3", "models/mllama", "models/ovis2", "models/paddleocr_vl", "models/paligemma", "models/perception_lm", "models/qwen2_5_vl", "models/qwen2_vl", "models/qwen3_5", "models/qwen3_5_moe", "models/qwen3_vl", "models/qwen3_vl_moe", "models/sam3", "models/sam3_tracker", "models/sam3_tracker_video", "models/shieldgemma2", "models/video_llama_3", "models/video_llava", "models/vipllava"] |
Cyrilvallez
left a comment
Cyrilvallez
left a comment
There was a problem hiding this comment.
Alright, just pushed a few changes to simplify the loading logic a bit!
LGTM otherwise, let's just wait for the slow tests as touching the mappings can silently break all the real-world weights!
| "timm_wrapper": [ | ||
| # Simply add the prefix `timm_model` | ||
| # TODO: Would be probably much cleaner with a `add_prefix` argument in WeightRenaming | ||
| WeightRenaming( | ||
| source_patterns=r"(.+)", | ||
| target_patterns=r"timm_model.\1", | ||
| ) | ||
| ], |
There was a problem hiding this comment.
Are you really really sure about this one? The base_model_prefix really covers it completely even when used as submodel etc?
CI ResultsCommit Info
The test failure analysis could not be completed. Please check the workflow run for details. |
|
run-slow: colmodernvbert, got_ocr2, paligemma, qwen2_5_vl, cohere2_vision, lfm2_vl, llava, shieldgemma2, sam3_tracker, paddleocr_vl, colpali |
|
This comment contains models: ["models/cohere2_vision", "models/colmodernvbert", "models/colpali", "models/got_ocr2", "models/lfm2_vl", "models/llava", "models/paddleocr_vl", "models/paligemma", "models/qwen2_5_vl", "models/sam3_tracker", "models/shieldgemma2"] |
|
run-slow: aria, aya_vision, cohere2_vision, colmodernvbert, colpali, colqwen2, emu3, ernie4_5_vl_moe, fast_vlm, florence2, fuyu, gemma3, gemma3n, glm46v, glm4v, glm4v_moe |
|
This comment contains models: ["models/aria", "models/aya_vision", "models/cohere2_vision", "models/colmodernvbert", "models/colpali", "models/colqwen2", "models/emu3", "models/ernie4_5_vl_moe", "models/fast_vlm", "models/florence2", "models/fuyu", "models/gemma3", "models/gemma3n", "models/glm46v", "models/glm4v", "models/glm4v_moe"] |
CI ResultsCommit Info
Model CI Report❌ 5 new failed tests from this PR 😭
|
|
Emu3 fails on main as well because of OOM, and locally the loading works without missing keys |
Cyrilvallez
left a comment
Cyrilvallez
left a comment
There was a problem hiding this comment.
Alright thanks! Let's keep an eye on tomorrow's slow CI run though to be sure!
|
[For maintainers] Suggested jobs to run (before merge) run-slow: aria, aya_vision, cohere2_vision, colmodernvbert, colpali, colqwen2, emu3, ernie4_5_vl_moe, fast_vlm, florence2, fuyu, gemma3, gemma3n, glm46v, glm4v, glm4v_moe |
|
View the CircleCI Test Summary for this PR: https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=44627&sha=90f253 |
3 participants
What does this PR do?
The diff in revert mapping is needed, otherwise we get failures in a few models, see https://app.circleci.com/pipelines/github/huggingface/transformers/167425/workflows/fa96efe5-f810-408e-bafd-de03b7e881aa/jobs/2208432/tests
The literal dot is a nice feat to have in general, we actually mean a literal dot in all of the patterns where
.is used. The being and end patterns are needed when reverse mapping from^model.language_modeltomodelbut the state dict has other keys such aspre_layer_proj_model. So it is reversed incorrectly if not marked explicitly as $ or ^