Convert notebook 05

[edbeeching]

This PR converts notebook 5, gpt2 sentiment control to work with new API.

I benchmarked for 2 hours - 200 iterations, the wandb report is here, here is the original one for comparison. (currently private, owned by @lvwerra )

Resolves issues #71 and #79

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lvwerra]

Hi @edbeeching, thanks for updating the notebook! Looks really good, here a few points:

• similar to the main sentiment notebook we could just use the pipeline for the classification and get rid of build_bert_batch_from_text
• we can indeed use accelerate for the device placement. ppo_trainer.accelerator.device should be where you get the device.
• if we replace sentiment_model.to(device) with sentiment_model.to(device); we can avoid printing the whole model graph
• I wonder if we should remove most of the logging from the loop for simplicity, what do you think?
• All the controlled continuation could be done much easier with the text-generation pipeline. I was young and didn't know any better :)

[younesbelkada]

This looks very good to me! Thanks for working on this !

[lvwerra]

There are still some issues regarding loss spikes (see #101). Will merge for now and investigate further. @natolambert you can use this notebook to see investigate the spikes. Full logs of a run can be found here.

[edbeeching]

@lvwerra and @younesbelkada , thanks for looking at, fixing and merging this. I have gone a bit silent due to pat leave, looking forward to getting back to work :)

[natolambert]

I am going to make another PR where this notebook is in example form -- much easier for doing multiple jobs and wider scale experimentation. It's also interesting that @edbeeching 's example didn't have the reward spike. I keep finding things to play with, so that's good for now.