[vlm] fix loading of retrieval VLMs

[zucchini-nlp]

What does this PR do?

As per title, reported internally that slow tests are failing. We need to apply same changes as in VLMs to the models that use VLMs in their architecture

[zucchini-nlp]

run-slow: colpali, colqwen2

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/colpali', 'models/colqwen2']
quantizations: [] ...

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

I wanted to use AutoModel as we shouldn't be loading the lm-head for these model. But the Qwen2-based model was released after refactor, and can work without any conversion_key_mapping currently, and I don't want us to add another key mapping just to use AutoModel instead of AutoModelForImageTextToText

[ydshieh]

Looks reasonable fix to me (as they seems to apply the same changes made to VLM)

@zucchini-nlp I observed that the 2 model tests fails in 2 different PRs, but maybe they share the same cause so their fix seems identical here?

For context:

colpali tests are failing after

[VLM] Add base model without head (#37033)

And for colqwen2, it fails after

[qwen] refactor attentions for vision/audio (#38930)

and there is a fix [qwen2-vl] fix vision attention scaling #39043, but that one doesn't fix for colqwen

[zucchini-nlp]

Hmm, ColQwen for me wasn't failing in a sense that the weights matched when laoding. But the tensors aren't close enough even after model was released. I can check out on runners and see what's the issue.

ColQwen shouldn't have the same issue, it was released after the major refactor

[ydshieh]

Hi, sorry, I think my memory got messed

#39043 (comment)

So that issue was already fixed, but my brain wasn't yet.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: colpali, colqwen2