Generate: slow assisted generation test

[gante]

What does this PR do?

test_assisted_decoding_matches_greedy_search fails once in a while, which blocks development. This PR removes the blocker by moving it to a slow test.

Why a slow test (and not redesign the test or add the flaky decorator)?

1. It is impossible to remove at 100% the non-determinism in this test. Some form of masking has to be used by design, which means that there is always a chance for the generations to diverge. When the generated sequences do diverge, the scores should be very similar at the step they diverge, as they are caused by the very small values within the numerical attention masks.
2. I've tried to add the check above (when the sequences diverge, the scores should be similar)... but some models still failed that check quite hard when the sequences didn't match. Some well-established models can run it without observed failures (e.g. 10k runs on GPT2 = 0 sequence mismatches). Others, especially roberta-based models, fail at a high rate. This means I should explore this further.
3. Since there is something to be explored, I believe the slow decorator is more appropriate: we can track failures without risking red CI on PRs.

[gante]

(this one was skipped because it was flaky)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ydshieh]

I know that we want to make PR CI clean, but I am not a super fan to see the daily CI report being flaky neither. It has been clear that it's hard to focus on the daily CI report, although things get a bit better with a recent PR to add diff between 2 PRs.

OK for me, and we can discuss/find a way to deal with this flaky situation 🔥 .

[ydshieh]

Just wondering, since this test uses batch size 1 and short sequence, could we run it 10 times and check the results with a ratio similar to #22996 ?

(well, it depends how fast is one test run however)

[gante]

The test is quite fast -- testing on all models takes ~14 seconds on my machine. With a low flaky ratio (<1% on average), repeating the test would not be expensive.

Nevertheless, since I found that some models are the cause, I'd like to defer repetition to after I explore the issue :D

[sgugger]

Does the error stay with non-random models? Maybe the test will be less flaky if done on a pretrained checkpoint (and since we are passing it at slow it would be ok to use those).

[sgugger]

Good for me to pass them as slow in any case, while investigating this further.

[gante]

Maybe the test will be less flaky if done on a pretrained checkpoint

Definitely! However, I think there is a deeper problem here, the logits diverge way more than I'd expect on some models, and it's odd that those models rely on the same base code (roberta). After I finish preparing the release for assisted generation, I'll get back to sorting related bugs