Add helpful and harmless example #128
Conversation
maxreciprocate
force-pushed
the
add-hh-example
branch
from
e695418
to
aedb7d7
Compare
maxreciprocate
force-pushed
the
add-hh-example
branch
from
4b93676
to
86e5e02
Compare
maxreciprocate
force-pushed
the
add-hh-example
branch
from
34817b6
to
a0894ea
Compare
| checkpoint_dir: "checkpoints/ppo_hh" | ||
|
|
||
| model: | ||
| model_path: "EleutherAI/gpt-j-6B" |
There was a problem hiding this comment.
Maybe let's make the default example use a smaller model so more people can run it
There was a problem hiding this comment.
Thanks, Max! I left a tiny nit for the README.md.
On another note, I'm hitting an issue when using the GPU-hosting launch command from the README.md (for both ILQL and PPO) where the samples list provided to the ppo_hh reward_fn is empty causing an IndexError. Could you please try this?
- Traceback Info: https://gist.github.com/jon-tow/e8aefbd0e7b06a1afc21dba38238d089
Note this only occurs when --num_processes is provided an odd number (quickly checked for 2, 4, and 6 processes fine).
|
@jon-tow regarding your issue, If you upgrade to |
There was a problem hiding this comment.
@reciprocated I had forgotten about the accelerate==0.15.0 metric bug 😅 This looks great from my end (not sure if @Dahoas has any more feedback).
This PR adds an example of training on Anthropic's dataset
PPO: https://wandb.ai/sorry/trlx/runs/ab0ehsxm
SFT: https://wandb.ai/sorry/trlx/runs/0aulb18k
ILQL: https://wandb.ai/sorry/trlx/runs/30ab7epk