Derive RewardModel from PreTrainedModel

[andreaskoepf]

First simple way to derive our RewardModel from PreTrainedModel so simplify loading .. still requires full download of base model which could probably be simplified in the future.

[sanagno]

LGTM

[dvruette]

Looks good! Does this work with AutoModel? If so, how exactly? I would expect it to work with AutoModelForSequenceClassification.from_pretrained.

Also, why not inherit from GPTNeoXModel? Could save us the trouble of loading the base model. We could basically just copy GPTNeoXForCausalLM and add pooling between hidden states and head.

[theblackcat102]

LGTM

[andreaskoepf]

Looks good! Does this work with AutoModel? If so, how exactly? I would expect it to work with AutoModelForSequenceClassification.from_pretrained.

This PR does currently not support loading from AutoModel but requires explicit use of the RewardModel class. If you have time you could look into what is necessary to register for AutoModel loading.

Also, why not inherit from GPTNeoXModel? Could save us the trouble of loading the base model. We could basically just copy GPTNeoXForCausalLM and add pooling between hidden states and head.

It isn't derived from a single architecture to allow different types of base models.