[t5] Fix negative kl issue

[younesbelkada]

Fixes #256

This PR fixes issues related with negative KL and T5 sentiment example. The first fix was related to the sentiment script that was incorrectly ported.

Before this PR, the padding side of tokenizers was always hardcoded to left in _batched_generate. in encoder-decoder models, they should be set to their native padding_side (i.e. right for T5), as the padding is performed on the encoder tokens, and these models have been trained with this specific padding side. I think that the culprit is the way the positional attention bias is computed, that does not take into account the starting position if padding_side=left: https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py#L436-L451

Thus forcing padding_side=left for encoder-decoder models should probably be avoided, as most of Enc-dec models pad the encoder input to the right (to verify).

To illustrate the fix:

• run on main with the modification on the example script only
• run on this branch

You can see that using padding_side=left led to unstable KL, whereas the proposed fix seems to lead to smoother KL

cc @lvwerra

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lvwerra]

maybe we could remove the special token here?

[younesbelkada]

I am not sure if this is correct as the token is removed here right after: https://github.com/lvwerra/trl/blob/ed87942a47f26d15e823ca7674737be02e48cc0a/trl/trainer/ppo_trainer.py#L832
Also made a run with generation[1:] and the KL becomes negative: https://wandb.ai/younesbelkada/trl/runs/vjbydeqv - so I think we shouldn't remove the special token here

[janpawlowskiof]

Is that not suspicious that such a slight change breaks the training? If no then why is that expected? I'm asking this as I myself am having trouble with my t5 model and negative kl divergence even on v.0.4.1 release.

[avacaondata]

With other encoder-decoder models such as MarianMT models (BART architecture) I'm still experiencing negative kl, in fact it is becoming increasingly negative: -5, -19, -24.

[Opdoop]

I experiencing negative kl warning with Alpaca-7B on the sentiment script.

[younesbelkada]

Can you try with non-batched generation as suggested in #256 (comment) and let us know if this works?

[lzy37ld]

In gpt2_sentiment.py, I have tried to modify the generation_kwargs a bit, like set the top_p to 0.9, and it said that my kl fall into negative and maybe because generation kwargs are not correctly set.

I wonder is there any specific restriction on the generation kwargs during ppo?

Gently pin:@younesbelkada