llava next image encoder to allow un-aligned patch / image sizes

[jimexist]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[Narsil]

Could you link to the problematic model so we can have a reference of what's expected and check that it's valid.

The assertion exists because it was supposed to be asserted as no such calculation is made in the original python code and drift is expected if it's violated.

[jimexist]

Could you link to the problematic model so we can have a reference of what's expected and check that it's valid.

The assertion exists because it was supposed to be asserted as no such calculation is made in the original python code and drift is expected if it's violated.

Thanks for the reminder

https://huggingface.co/google/siglip-so400m-patch14-384/discussions/4

This is the discussion link and the model is siglip so400m

[jimexist]

@Narsil let me know if this is an acceptable approach.
another solution is to allow the check to be by-passed given args or env var. but i guess it's more invasive to the codebase

[jimexist]

hi @Narsil any possibility this PR gets merged?

[danieldk]

Thanks for the reminder

https://huggingface.co/google/siglip-so400m-patch14-384/discussions/4

This is the discussion link and the model is siglip so400m

I am trying to reproduce this issue, but the linked model uses siglip as the model type (which we don't support), but this PR changes the handling of llava_next. Could you give more information on how to reproduce the issue?