Add CPOTrainer

[fe1ixxu]

Hi! This PR wants to add CPOTrainer proposed in the paper Contrastive Preference Optimization: Pushing the Boundaries of LLM
Performance in Machine Translation

The CPO method is one of the algorithm for building the state-of-the-art LLM-based translation model: ALMA

[fe1ixxu]

cc @kashif @lewtun

[kashif]

@fe1ixxu how close is the trainer in terms of code to the DPOTrainer? Can one subclass from it?

[fe1ixxu]

@kashif Thanks for the quick response! CPO is an approximation of DPO. The key differences between CPOTrainer and DPOTrainer are:

• Remove the need of reference model
• add an extra NLL loss for preferred data

I'm uncertain whether subclassing CPOTrainer from DPOTrainer is a proper idea, as DPOTrainer introduces numerous features related to reference models that are unnecessary for CPOTrainer.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[fe1ixxu]

Hi @kashif CPO docs has been finished now! Thanks!

[fe1ixxu]

Hi @kashif and @lewtun, I see implementation of CPOTrainer has been finished for a while and it has passed all checks. If this looks good to you, is there any chance to merge it to the main branch? Thanks!

[lewtun]

Thank you for this very nice implementation of CPO @fe1ixxu 🔥 ! I left a few small comments and a suggestion to remove a deepspeed function I don't think we need. Apart from that LGTM!