fix florence kwargs

[SunMarc]

What does this PR do?

This PR fixes Florence 2 model kwargs issue with num_items_per_batch. Since BART decoder don't take any kwargs, we shouldn't pass them in forward, otherwise we get an error.
TypeError: BartDecoder.forward() got an unexpected keyword argument 'num_items_in_batch'

cc @merveenoyan

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: florence2

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[merveenoyan]

I tested it and it fixes my issue, thanks a ton!

[vasqu]

Wouldn't it be better to allow loss kwargs on Bart (top level)?

The reason Bart doesn't have the kwargs support is that while the attentions are already refactored. there are 3 different attention flavors (encoder, decoder, cross) which needs a proper separation (not covered by our current way of things). And this issue seems specific to just the loss calculation so should be fine to add kwargs on the user facing modules and type it properly.

[SunMarc]

I agree with what you said @vasqu. Maybe we can first merge this PR to unblock @merveenoyan and do that in a follow-up PR as this PR doesn't create any regression?

[vasqu]

Gotcha, please update this later then 🤗 since it doesn't have attention backend support (yet), it should be fine