[DPO] remove response/pairs from the DPO side

[kashif]

fixes #537

TODO:

• docs
• tests

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[tomaarsen]

This function can be simplified a lot now:

def get_hh(split: str, sanity_check: bool = False, silent: bool = False, cache_dir: str = None) -> Dataset:
    """Load the Anthropic Helpful-Harmless dataset from Hugging Face and convert it to the necessary format.

    The dataset is converted to a dictionary with the following structure:
    {
        'prompt': List[str],
        'chosen': List[str],
        'rejected': List[str],
    }

    Prompts should be structured as follows:
      \n\nHuman: <prompt>\n\nAssistant:
    Multiple turns are allowed, but the prompt should always start with \n\nHuman: and end with \n\nAssistant:.
    """
    dataset = load_dataset("Anthropic/hh-rlhf", split=split, cache_dir=cache_dir)
    if sanity_check:
        dataset = dataset.select(range(min(len(dataset), 1000)))

    def split_prompt_and_responses(sample) -> Dict[str, str]:
        prompt = extract_anthropic_prompt(sample["chosen"])
        return {
            "prompt": prompt,
            "chosen": sample["chosen"][len(prompt) :],
            "rejected": sample["rejected"][len(prompt) :],
        }

    return dataset.map(split_prompt_and_responses)

I can't push this to the PR directly or add this as a suggestion in the review, sadly.

[kashif]

you should be able to push to my branch now... please feel free!

[kashif]

let me fix up the docs and test

[kashif]

@younesbelkada or @lvwerra should be ready for review!

[younesbelkada]

Looks good on my side, thanks a lot everyone for working on this!
Since we did not release a pypi version with DPOTrainer the breaking change is ok to have it now!

[tomaarsen]

I appreciate that you left some time for people to find these issues before releasing. I'd love to look into the spamminess of the logs during training as well. Do you know when you intend to release?

• Tom Aarsen

[kashif]

yes @tomaarsen let's open another PR for the logging issue... ideally, I wanted to have this all logged to wandb etc. and note we also have the sample generation helper too which is currently not being used...

[tomaarsen]

wandb support should be automatically implemented right? As long as it's installed and the report_to is set up right.
Is the issue simply that we don't call self.callback_handler.on_log(self.args, self.state, self.control, ...)?

I can help with this, as I have a good amount of experience with how Transformers does its Trainer via a SetFit logging/callbacks integration that I've been working on.

• Tom Aarsen

[kashif]

yup! for some reason i was not getting it to log with the trainer and so I "forced" it to log things via the call to:

if self.accelerator.is_main_process:
     self.log_metrics("test", metrics)

would appreciate any insight here!

[younesbelkada]

Thanks everyone! For the release probably mid-next week would be a nice time, but not sure sure @lvwerra