🚨 Allow check_model_inputs in core VLMs

[zucchini-nlp]

What does this PR do?

Unblocks #39722 so we can use check_model_inputs in VLMs copied from llava. Otehrwise #39722 would need to re-define model classes to change forward pass and delete output_xxx kwargs

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qubvel]

Thanks, looks very nice! Did you check if that's breaking in terms of actually returned hidden_states? They might be different, especially for the last item in the list. Still worth unbloating, but we might need to add 🚨

[zucchini-nlp]

Did you check if that's breaking in terms of actually returned hidden_states

Checked the test_hidden_states and test_attentions which checks only sizes and lengths. I will trigger check with an actual model for values as well, to be sure it works

[zucchini-nlp]

run-slow: aya_vision, got_ocr2, internvl, llava, mistral3, perception_lm, vipllava

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/aya_vision', 'models/got_ocr2', 'models/internvl', 'models/llava', 'models/mistral3', 'models/perception_lm', 'models/vipllava']
quantizations: [] ...

[qubvel]

Thanks! I still see output_attentions/output_hidden_states/return_dict in modeling files, but suppose they should gone entirely in case you enable check_model_inputs. Am I missing smth?

[zucchini-nlp]

not all models have a last_hidden_state. In some multimodals, we return only vision_tower.hidden_states or similar and thus cropping the last hidden is not correct. We want to crop it only if it is being replaced right away with a hidden state after final "layer_norm" (e.g llama)

[zucchini-nlp]

run-slow: aimv2, aya_vision, blip, blip_2, got_ocr2, idefics, idefics2, idefics3, instructblip, instructblipvideo, internvl, janus, llava, mistral3, ovis2, phi4_multimodal

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/aimv2', 'models/aya_vision', 'models/blip', 'models/blip_2', 'models/got_ocr2', 'models/idefics', 'models/idefics2', 'models/idefics3', 'models/instructblip', 'models/instructblipvideo', 'models/internvl', 'models/janus', 'models/llava', 'models/mistral3', 'models/ovis2', 'models/phi4_multimodal']
quantizations: [] ...

[zucchini-nlp]

Should be ready now, updated all VLMs (+ had to update Siglip from which some model copy). In base models which consist of backbones, I didn't add any check_model_inputs decorator since outputs will be handled by each backbone and we just need to pass forward what the backbone returned

Tests are very flaky, so CI will be red

[zucchini-nlp]

run-slow: aimv2, aya_vision, blip, blip_2, got_ocr2, idefics, idefics2, idefics3, instructblip, instructblipvideo, internvl, janus, llava, mistral3, ovis2, smolvlm

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/aimv2', 'models/aya_vision', 'models/blip', 'models/blip_2', 'models/got_ocr2', 'models/idefics', 'models/idefics2', 'models/idefics3', 'models/instructblip', 'models/instructblipvideo', 'models/internvl', 'models/janus', 'models/llava', 'models/mistral3', 'models/ovis2', 'models/smolvlm']
quantizations: [] ...

[qubvel]

Very nice! I did not review every line, but the main idea is to double-check return type hints to make sure they match with the changed outputs

[qubvel]

Thanks for the update, looks good to me!

This one might be relevant to merge to make sure users are aware if they request output_attentions=True

• #40597

Also, for Siglip (and later for CLIP), it might be worthwhile to keep hidden_states collection explicit because that's something natural for vision models to output for many downstream tasks, such as segmentation. As far as I understand, CLIP hidden_states are used in diffusers as well. I did it partially in

• #40505

can rebase on your version if it would be merged earlier

[zucchini-nlp]

Oke, ig all loading issues are resolved in main, so rebase will help with flakiness once and for all

[zucchini-nlp]

Why is it flaky, it downloads with hf hub 😭

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: aimv2, altclip, aya_vision, blip, blip_2, clip, clipseg, git, got_ocr2, idefics, idefics2, idefics3, instructblip, instructblipvideo, internvl, janus