-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Padding side incorrect for Mistral DPO #1217
Comments
Did some debugging, quiet confirmed that https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L538C4-L550 gives incorrect padding side and lead to crash in transformers.
|
Can I take this up @vwxyzjn? |
I had the same issue and went down a debugging rabbit hole, but realised in the end I had forgotten to set Remember you need to set |
Checked this solution work. |
Hi thanks for the issue. Maybe during training the tokenizer.padding_side should be set to right? |
Any reason that training should be right? For most of dpo scripts I have seen, they all set mistral tokenizer padding to left.
|
https://towardsdatascience.com/fine-tune-a-mistral-7b-model-with-direct-preference-optimization-708042745aac Does setting pad_side to left impact DPO result? What is the correct setting we should use? |
This should be now fixed on TRL main ! |
For code here: https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L538C4-L550 Does it always padding to right?
I get error when run dpo for mixtral 7b.
I am still seeing this error after set
tokenizer.padding_side = 'left'
This is my demo notebook: https://colab.research.google.com/drive/1sVqbYEOqjJYl7CzNzXzviEBB6A984cMq?usp=sharing
Tokenizer already set with left padding
![截屏2024-01-10 下午10 53 07](https://private-user-images.githubusercontent.com/25971665/295606981-0ce9f431-8a8f-4236-8c71-846f183f455e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MTc1ODUsIm5iZiI6MTcyNDQxNzI4NSwicGF0aCI6Ii8yNTk3MTY2NS8yOTU2MDY5ODEtMGNlOWY0MzEtOGE4Zi00MjM2LThjNzEtODQ2ZjE4M2Y0NTVlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIzVDEyNDgwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRiOGZlMTUxYWVjODczYjZiZmIxM2RhYTJhZWZiYTBmZGUwMTgyNWFjNTAzMjA2ZDg4YWIxNjQxNTE2Yzk5M2QmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ChAiT2xDUAvdXS18lr9rGbC-PXLgLF4fvwk3_MGvuI4)
Train
![截屏2024-01-11 上午10 28 53](https://private-user-images.githubusercontent.com/25971665/295769784-e8f0a944-9f22-4cba-88a6-a1712ba02f2c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MTc1ODUsIm5iZiI6MTcyNDQxNzI4NSwicGF0aCI6Ii8yNTk3MTY2NS8yOTU3Njk3ODQtZThmMGE5NDQtOWYyMi00Y2JhLTg4YTYtYTE3MTJiYTAyZjJjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIzVDEyNDgwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY1NWQzNmNmNjY5Njc2ZDNjZTIyNjE4NzhlYTdhYWQ0NmM2ZTM3MDBjNTJlZTc5NDg1NmQ5Yzc2MWU5MWMxOTUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.6uqJQERJcVWr17b1qKLVVdD2T5ZZxl0W0B2YiLsxXrg)
Still have:
ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to call
tokenizer.padding_side = 'left'before tokenizing the input.
Seems https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L538C4-L550 or other part of code might lead to bug?
transformers 4.36.2
trl 0.7.7
peft 0.6.0
The text was updated successfully, but these errors were encountered: