ORPOTrainer fails with flash-attention2

[alvarobartt]

Description

The ORPOTrainer fails when training using attn_implementation="flash_attention_2", since the cache is being used, and falls back to the default configuration i.e. padding_side="right" for the tokenizer in this case.

Bug in code

Missing use_cache=False to prevent the model from using the cache, to avoid issues with Flash Attention 2.

trl/trl/trainer/orpo_trainer.py

Lines 685 to 689 in 2ce8e45

	outputs = model(
	concatenated_batch["concatenated_input_ids"],
	attention_mask=concatenated_batch["concatenated_attention_mask"],
	**model_kwargs,
	)