ORPO trainer

[kashif]

ORPO trainer

Reference-free Monolithic Preference Optimization with Odds Ratio

cc @jiwooya1000

• figure out what to log
• add logging section to the docs

[kashif]

cc @philschmid

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lewtun]

Very cool to see this elegant method get added so quickly @kashif !

I left a few remarks about what to log etc and to harmonise the example script to be less hard-coded wrt Anthropic HH. I'd also like to see some small experiments which show the metrics look sane for the examples. Otherwise it's looking great!

[lewtun]

WDYT about logging the log probs and log odds ratio alongside the SFT loss, OR loss and full loss? This way the user can debug if the rejected log probs are decreasing over the course of training

[lewtun]

I think this could live in the ORPOConfig (ideally we want nearly everything that is not a callable to live in a single config so it can be easily tweaked at the command line)

[lewtun]

@kashif WDYT about this proposal to move the arg to the config?

[lewtun]

Maybe we can just keep logps and nll_loss and also log odds ratio to simplify this?

[jiwooya1000]

Thank you so much for such a fast implementation of ORPO @kashif😀

Also, about #1435 (comment), I think it is a great idea to log the log odds ratio as it can help monitor the effect of β(the weighting hyperparam) as this report.

Thank you again for the implementation!

[kashif]

ok i'll add the logging next

[vwxyzjn]

Great work! Added some comments.

[vwxyzjn]

We now have standard datasets under https://huggingface.co/datasets/trl-internal-testing/hh-rlhf-trl-style (#1424).

• https://github.com/huggingface/trl/blob/main/examples/datasets/anthropic_hh.py creates the standard dataset (prompt, chosen, rejected)
• https://github.com/huggingface/trl/blob/main/examples/datasets/tokenize_ds.py is a usage example.

In this case, maybe you could try:

    ds = load_dataset(args.dataset)
    if args.debug:
        for key in ds:
            ds[key] = ds[key].select(range(50))
    tokenizer = AutoTokenizer.from_pretrained(args.model)
    if tokenizer.chat_template is None:
        tokenizer.chat_template = "{% for message in messages %}{{message['role'] + ': ' + message['content'] + '\n\n'}}{% endfor %}{{ eos_token }}"

    def process(row):
        row["chosen"] = tokenizer.apply_chat_template(row["chosen"], tokenize=False)
        row["rejected"] = tokenizer.apply_chat_template(row["rejected"], tokenize=False)
        return row

    ds = ds.map(
        process,
        num_proc=1 if args.debug else multiprocessing.cpu_count(),
        load_from_cache_file=False,
    )
    train_dataset = ds["train"]
    eval_dataset = ds["test"]
    trainer = ORPOTrainer(
        model,
        args=orpo_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        tokenizer=tokenizer,
        peft_config=get_peft_config(model_config),
    )

here the args.debug is doing the same thing as sanity_check.

[lewtun]

Thanks for iterating @kashif ! I left one final nit and a question about moving the dataset proc args to the config. Apart form that LGTM 🔥

[lewtun]

@kashif WDYT about this proposal to move the arg to the config?

[jiwooya1000]

Just gave a quick try with ORPOTrainer with facebook/opt-350m + argilla/ultrafeedback-binarized-preferences-cleaned, and it seems to be working well😃 Thank you for your work @kashif!

[alvarobartt]

Maybe it would also be nice to include the ORPOTrainer within the README.md listing of supported trainers see https://github.com/huggingface/trl?tab=readme-ov-file#highlights

[kashif]

@alvarobartt ok yes good idea! adding