Feature/remove reward instructor

[CloseChoice]

closes #2049

I updated the model_training/utils.py with all the functionality I could find diverging from reward/instructor.

I still have a few questions:

• do we need the all of the dataset processing functionality form reward/instructor, e.g. in experimental_dataset.py and cls_dataset.py and alsow the webgpt_return_format function in utils?
• will we use webgpt and hfsummary as reward model training data? Currently only the oasst_export data is defined as training data for the RM model

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[olliestanley]

In terms of checking whether we need functionality from reward/instructor, I wouldn't worry too much. If any of it gets deleted and turns out to be needed later, it's still in the Git history so easy enough to go back and grab! I'm also not sure if we need to add all those configs to the YAML as most of them are probably outdated now but will leave that up to the ML guys to review

[andreaskoepf]

@CloseChoice since you are actively developing in the ML parts of open-assistiant I would like to invite you to join the OA ML team on discord. Please ping me (andreaskoepf).

[dvruette]

Changes look good. Not sure we actually still need the old RM configs, but it's probably fine to keep them around for a little longer.

thx!

[andreaskoepf]

@theblackcat102 Is all relevant RM code already part of the new trainer_rm so that the old reward/instructor can be deleted? At least the old rank datasets were not ported yet. What's with rankgen loss etc?

[CloseChoice]

Do we have a dataset with which I can check if that works correctly? I don't have access to the private OA data but I guess that a any other dataset suitable for RM should suffice. Even if I have to manipulate it a bit.

[theblackcat102]

@andreaskoepf rankgen didn't have good results to back it up anyway. I think its fine if we just move on

[theblackcat102]

@CloseChoice I saw you added RM configs for deberta, however the existing code doesn't work for deberta nor due to limited choices in TOKENIZER_CONFIGS under utils.py. Have you tried running a webgpt examples on these new RM config?

[CloseChoice]

@theblackcat102 I tried, but so far I failed. Currently we only support oasst_export as RM dataset (as given by the RM_DATASETS and I don't see any reward modelling functionality in the WebGPT dataset (and don't see it under rank_datasets aswell). Should WebGPT just work out-of-the-box if I add it to RM_DATASETS?

I just added the configs from reward/instructor and copied them as they were.

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[CloseChoice]

I fixed the HFSummary dataset and ran

python trainer_rm.py --configs defaults_rm debug_rm

with this branch and ran into the following error

  File "xxx/Open-Assistant/model/.venv/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 210, in _unscale_grads_
    raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.

I'm not the first to have this problem but any ideas how to fix this?

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[CloseChoice]

I removed the configs where I ran into problems with the tokenizer config

[theblackcat102]

@CloseChoice the problem with webgpt, hf_summary and ValueError: Attempting to unscale FP16 gradients. are all fixed here Zzzzz

[CloseChoice]

@theblackcat102 still get the errors on main

[CloseChoice]

@theblackcat102 @andreaskoepf @dvruette Any reasons why we should not merge this?

[theblackcat102]

@shahules786 is this a bug?

        for data in dataset:
            for item in dataset:

[CloseChoice]

yes, that is a bug I fixed on the fly

[theblackcat102]

LGTM