Padding side incorrect for Mistral DPO

[hengjiUSTC]

For code here: https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L538C4-L550 Does it always padding to right?

I get error when run dpo for mixtral 7b.

I am still seeing this error after set tokenizer.padding_side = 'left'
This is my demo notebook: https://colab.research.google.com/drive/1sVqbYEOqjJYl7CzNzXzviEBB6A984cMq?usp=sharing

Tokenizer already set with left padding

Train

Still have: ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to call tokenizer.padding_side = 'left' before tokenizing the input.

Seems https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L538C4-L550 or other part of code might lead to bug?

transformers 4.36.2
trl 0.7.7
peft 0.6.0

[hengjiUSTC]

Did some debugging, quiet confirmed that https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L538C4-L550 gives incorrect padding side and lead to crash in transformers.

File ~/learn-llm/venv/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py:899, in MistralModel.forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
    897     is_padding_right = attention_mask[:, -1].sum().item() != batch_size
    898     if is_padding_right:
--> 899         raise ValueError(
    900             "You are attempting to perform batched generation with padding_side='right'"
    901             " this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to "
    902             " call `tokenizer.padding_side  = 'left'` before tokenizing the input. "
    903         )
    905 if self._use_flash_attention_2:
    906     # 2d mask is passed through the layers
    907     attention_mask = attention_mask if (attention_mask is not None and 0 in attention_mask) else None

ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to  call `tokenizer.padding_side  = 'left'` before tokenizing the input. 

[gchhablani]

Can I take this up @vwxyzjn?

[arkapal3]

I had the same issue and went down a debugging rabbit hole, but realised in the end I had forgotten to set use_cache to False for my reference model.

Remember you need to set use_cache = False for both the main and reference model. If you do so the check on is_padding_right above won't trigger (which it shouldn't, because we are using the forward to compute log probs, not do generation).

[hengjiUSTC]

I had the same issue and went down a debugging rabbit hole, but realised in the end I had forgotten to set use_cache to False for my reference model.

Remember you need to set use_cache = False for both the main and reference model. If you do so the check on is_padding_right above won't trigger (which it shouldn't, because we are using the forward to compute log probs, not do generation).

Checked this solution work.

[vwxyzjn]

Hi thanks for the issue. Maybe during training the tokenizer.padding_side should be set to right?

[hengjiUSTC]

Any reason that training should be right? For most of dpo scripts I have seen, they all set mistral tokenizer padding to left.

Hi thanks for the issue. Maybe during training the tokenizer.padding_side should be set to right?

[vwxyzjn]

For both our and DPO's original repo, we all do right padding. Out of curiosity, which DPO scripts have you seen use the pad left? Pad left is primarily used for generation such as in PPO but in DPO it's unnecessary because DPO does not do generation.

[hengjiUSTC]

https://towardsdatascience.com/fine-tune-a-mistral-7b-model-with-direct-preference-optimization-708042745aac
https://colab.research.google.com/drive/15iFBr1xWgztXvhrj5I9fBv20c7CFOPBE?usp=sharing

Does setting pad_side to left impact DPO result? What is the correct setting we should use?

[younesbelkada]

This should be now fixed on TRL main !