Learning to Summarize Reward Model Loss indexing

[natolambert]

Been improving on this for H4, and I think there may be a slight bug. At each reward calculation, which input has a higher reward needs to be known to pass it into the model.

In TRL: https://github.com/lvwerra/trl/blob/a05ddbdd836d3217c80a4b3e679ba984bfd4fa24/examples/summarization/scripts/reward_summarization.py#L185

Here's the original paper, note how the indexing depends on which is selected. Or, maybe this is handled elsewhere in the script (I didn't see it).

[natolambert]

FYI @TristanThrush you may know this :)

[lvwerra]

Isn't this handled here: https://github.com/lvwerra/trl/blob/f95be7736fcb4eff59964c8857a4fa8e05fe2632/examples/summarization/scripts/reward_summarization.py#L97-L102

[TristanThrush]

In this code, it is assumed that the j examples are the preferred ones and the k examples are not preferred.

https://github.com/lvwerra/trl/blob/a05ddbdd836d3217c80a4b3e679ba984bfd4fa24/examples/summarization/scripts/reward_summarization.py#L90

Let me know if I misunderstood anything

[lvwerra]

Makes sense to me. @natolambert?

[natolambert]

Yup, took me a while to find the comments and that's right. May get more complicated in the future, but my deep dive result is: this code looks right.