Do not erase a cache_position passed explicitly to generate(), if there is one

[FremyCompany]

What does this PR do?

Currently, any cache_position passed to model.generate() is silently replaced by the initial initialization. This seems wrong, since there is no need to reinitialize an argument that has already been passed. In some models, being able to pass custom cache_positions which are not 0-initialized is required for proper behavior.

This PR adds a check to the initialization code of model_kwargs['cache_position'] to return early if that argument was already provided.

There is no linked issue because I fixed the bug before filing an issue. PR is thus easier.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

I don't think that this warrants any documentation change, the current behavior is not described anywhere. I could add a test, but it would be very artificial. I think merging as-is is fine, but I welcome feedback.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @zucchini-nlp @gante

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

[zucchini-nlp]

Thanks for the PR! I am curious, is the logic to create build positions from scratch not working in some cases?

Not many users know about cache_position or pass a custom new position, so I guess this is for a custom usage with compressed cache

[FremyCompany]

The use case is related to Mamba models, so the logic for initializing the cache position unfortunately doesn't work because there is no way to know how many tokens were already processed, and starting with index 0 disables the cache since it is supposed not to be initialized yet for the first token.

[FremyCompany]

Calling generate is the easiest way to continue generating after the first batch of tokens. Calling forward directly would also work but then it's necessary to reimplement all the sampling stuff which generate usually does for you in a very convenient way.

[zucchini-nlp]

Thanks for explaining, makes sense. For special models we don't infer cache position correctly. Can you makr the PR ready for review and rebase main, so it can be merged?

[FremyCompany]

Yes, will do tomorrow.

[gante]

👌

Thank you for noticing it!

[gante]

@FremyCompany to make our CI happy:

• Tag the PR as Ready for review (button at the bottom of the PR page)
• run make fixup on your terminal, inside the transformers root folder (this runs automated code formatting)
• commit the changes

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.