Added RM dataset training split and add support for WebGPT and other for RLHF

[theblackcat102]

1. Added the code I use for splitting RM dataset : tools/sample_rm_data.py

python -m tools.sample_rm_data 2023-02-12_oasst_prod.jsonl

This generates 3 splits of rm_test.jsonl, rm_train.jsonl, rm_val.jsonl

2. Added webgpt and private_tuning for RLHF dataset

    - oa_private:
        data_path: .cache
        split: rl
        val_split: 0.0
        fraction: 1
        file: 2023-02-12_oasst_prod.jsonl
    - webgpt:
        val_split: 0.0
        fraction: 1
    - private_tuning:
        val_split: 0.0
        fraction: 1

make sure oa_v3_fixed_plus_safety.jsonl used in private_tuning is placed under .cache

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[theblackcat102]

Some new todos:

1. we might have to only sample the english only prompts, cause spanish was around 29% and current pythia model might perform pretty bad in spanish?

en: 57.2%
es: 28.97%
...

2. Add a reweight process to ensure the initial mean reward score starts at zero mean.

Previous snippet code I written

[sanagno]

Great, thanks!