Online DPO Support

[dhineshkumar-r]

Hey Folks,

Does Nemo-RL support Online DPO(https://arxiv.org/abs/2402.04792)?
Given there are environments in Nemo-RL, was wondering about the possibility of generating preferences online based on the outcomes in the environment and fine-tuning.

[guyueh1]

@terrykong could you estimate the effort of this and consider if we should put it in roadmap?
@snowmanwwg to comment on community reflected importance

[terrykong]

Would love to know if others would be interested in this feature.

The effort would mainly be re-using components in grpo.py to construct this loop (and then use the DPO loss). It looks like if you use an LLM annotator, it would be like having a genRM in the loop, but from the paper, unless i missed it, it looks like the genRM could just be the same as the policy you are using, and they just do a stop gradient on it.