Does DPOTrainer loss mask the prompts?

[tokestermw]

Hi quick question, so DataCollatorForCompletionOnlyLM will train only on the responses by loss masking the prompts.

Does it work this way with DPOTrainer (DPODataCollatorWithPadding) as well? Looking at the code, it does look like it trains on the prompts. But maybe it doesn't matter with the DPO loss.

The reason I ask is my dataset has long prompts, but short responses. And the resulting model trained by DPO is barely different from the reference model (even with accuracy 90+%). So loss masking the prompts may help focus the learning, just a guess.

Thanks!

[lvwerra]

As far as I know we don't mask prompts in DPO but @kashif might know more about this.

[rpowalski]

I am having the same problem with the DPO training not improving the baseline model

Nevertheless, DPODataCollatorWithPadding seems to put an ignore index for label indexes corresponding to prompt so probably the reason is elsewhere
Here is the relevant piece of code that does that:

trl/trl/trainer/utils.py

Line 384 in baa8f09

chosen_sequence_tokens["labels"][: len(prompt_tokens["input_ids"])] = [self.label_pad_token_id] * len(

[tokestermw]

ah thanks! @rpowalski

[sadaisystems]

Gentlemen, If I understood your discussion correctly (and the code snippet mentioned), DPOTrainer does in fact mask out the prompt tokens by default? Were you able to figure out why your models did not improve?

@rpowalski @tokestermw