Mistral with flash attention 2 and right padding

[dakinggg]

System Info

• transformers version: 4.34.0
• Platform: Linux-5.4.0-148-generic-x86_64-with-glibc2.31
• Python version: 3.10.13
• Huggingface_hub version: 0.17.3
• Safetensors version: 0.4.0
• Accelerate version: 0.20.3
• Accelerate config: not found
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed

Who can help?

@younesbelkada

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

If you run a batch through mistral with flash attention 2 with right padding, you get

ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to  call `tokenizer.padding_side  = 'left'` before tokenizing the 
input.

I am not doing generation, just calling forward. Is the error message incorrect and you actually meant to prevent all usage of right padding here? Or is the implementation wrong and this was meant to only prevent generate usage of right padding? Or perhaps I am missing something else. Thanks!

Expected behavior

Either right padding is ok for calling forward, or the error message correctly states the problem.

[imraviagrawal]

Having the same issues
ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to call tokenizer.padding_side = 'left' before tokenizing the input.

[ArthurZucker]

cc @younesbelkada

[younesbelkada]

Indeed forward should be supported but not generation, will raise a patch soon for this

[hengjiUSTC]

Hey @younesbelkada, I am still seeing this error after set tokenizer.padding_side = 'left'
This is my demo notebook: https://colab.research.google.com/drive/1sVqbYEOqjJYl7CzNzXzviEBB6A984cMq?usp=sharing

Tokenizer already set with left padding

Still have: ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to call tokenizer.padding_side = 'left' before tokenizing the input.

Not sure is this because of trl or something wrong within transformer?

transformers 4.36.2
trl 0.7.7
peft 0.6.0

[wenhuchen]

I am having the same issue. Even after I set the tokenizer padding side = left, this error still occurs during training.

[younesbelkada]

Thanks everyone for reporting, might be an issue with TRL I think, let me have a deeper look and get back ASAP

[hengjiUSTC]

Thanks everyone for reporting, might be an issue with TRL I think, let me have a deeper look and get back ASAP

I opened a issue in trl as well: huggingface/trl#1217 (comment)

[arkapal3]

You need to set use_cache = False for both the main and reference model. See my comment here: huggingface/trl#1217 (comment)

[nguyen-brat]

You need to set use_cache = False for both the main and reference model. See my comment here: huggingface/trl#1217 (comment)

I try your solution and it work like a charm but does set use_cache to False make tokenizer.padding_side = 'left' during evaluation and 'right' during training. I read the doc about use cache here: link but seem like it just reduce the inference time. Can you explain why it work like a charm sir ?