Fix siglip flaky test_eager_matches_sdpa_inference

[ydshieh]

What does this PR do?

This test is very flaky for dtype bf16 for 3 reasons:

• bf16 has bad precision, especially when the magnitude is large
• siglip has larger outputs (than llama, clip, for example). The maximal abs. values for them:
  • siglip: 3.0~4.0
  • clip: 0.9 ~ 1.0
  • llama: ~0.1
• siglip tests have sequence length 225, i.e. num_patches = (image_size: 30 / pat
• ch_size: 2) ** 2

All 3 combined cause it's much more flaky.

This PR address this.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

even with this (retry 5 times), we still had some failures from time to time.

Now we don't need this anymore

[zucchini-nlp]

yeah, i think with rtol we can find one value that works for siglip and llama

[ydshieh]

running this (parameterized) tests 1000 times (for each set of parameters) for siglip, no failure. 😢

(but yeah, the comment here is more for future weird models :-) )

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: siglip

[ydshieh]

It turns out the very small hidden dimension will cause some edges cases and fail this test for other cases like logits_per_text.
I just revert the changes but only keep the image_size very small (i.e. sequence length small)

Works for all cases now.