Refactor rlhf

[maxjeblick]

This PR adds a separate problem type for RLHF.

Some discussion items:

• How to structure the training settings configuration. The current order may not be optimal.
• train.py contains two training functions (run_train_rlhf and run_train) with partially duplicated code. IMO, it is ok to keep it as is rather than having one function with multiple if-else statements.
• Train data insights should be redone (Target Text is not used, thus always an empty string). Will implement it here after Max/insights table view #301 has been merged.

Fixes #317
We can first merge #308 and I will fix potential merge conflicts subsequently.

[maxjeblick]

Should be good for a first review.

Some points I noticed:

• Train data insights do not display the answer text. We could either leave it as is (highlighting that training is not using ground truth answer), or removing the field by subclassing the plot class.
• Some inference parameters for RLHF are hidden in the chat window. One could explicitly change the visibility in the chat tab?

[pascal-pfeiffer]

Thanks a lot @maxjeblick .
I really like your refactors and it makes great sense to put RLHF in an own problem type.

I will still need to go through the model and train changes and also do some local testing, but will already provide you some initial thoughts now.

[pascal-pfeiffer]

• LLM Backbone is not a dropdown for me

• hide output dir

We could probably move the reward model to the top, too, wdyt?

[pascal-pfeiffer]

and let's also remove

• FSDP
• skip parent proba
• random parent proba

[maxjeblick]

Thanks for the review, it was very helpful. I addressed the issues above, in addition I added some smaller code changes:

• I fixed a small bug in plot_batch
• I refactored .from_dict class method to a common base method.

We could probably move the reward model to the top, too, wdyt?

Yes good idea, I changed that.

[pascal-pfeiffer]

Thanks a lot @maxjeblick , very nice refactor and small fixes along the way!
Looks all good to me now.

I will likely work on RLHF again in a follow up PR/issue and allow for larger batches. It should be much easier now with the individual train loops.