Add support for GPT-neo and GPT-J

[chijames]

Hi,

Thanks for the great project! Do you plan to support GPT-neo and GPT-J down the road?

[ShengdingHu]

Thanks for your comment, the project is basically compatible with any PyTorch-based models.
To apply OpenDelta to a new model that we haven't tested, please follow the instructions in the docs, basically,

1. visualize and understand the parameter/modular structure of the pre-trained model using Visualization
2. specify the modified modules according to your knowledge about the delta methods, e.g., if applying Lora, any linear can be specified, if applying adapters, any submodules that have hidden state as the output can be specified.
3. freeze the irrelevant parameters.

However, as a kind reminder, prefix-tuning is not supported due to its compicated nature to process the attention mask.

If you still have any other questions, feel free to let us know.