Extend AdaptionPrompt and Add Multi-Modal AdaptionPromptV2

[PanQiWei]

What does this PR to

This PR:

• extends AdaptionPrompt to support gptj, gpt-neox and moss and can be futher extended to more models in the future;
• add a multi-modal version AdaptionPromptV2 implementation that has a slightly modification to the original one proposed in Support More Models for Adaption Prompt and Add Adaption Prompt V2 with multi-modal ability supportion #398 so that people can training on multi-modal dataset (such as coco, reference project), note that the implementation of multi-modal functionality may different to the official code of llama_adapter_v2.

@pacman100 @younesbelkada if this pr look good to you guys, I will close #398 and keep working on this one to make it merge-able 🙏

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[BenjaminBossan]

Hi, thanks for you continued work. Could you please run make style so that the code quality checks passes and the rest of the CI can run?

[PanQiWei]

The tests are passed, I will futher improve documentation and add some more tests, then I think this pr can be reviewd ❤️‍🔥

[BenjaminBossan]

I'm wondering: Would it be a lot of work for you to split the two changes you suggested into separate PRs? That would make it easier for us to review the changes.

[PanQiWei]

Not at all, I will do it this weekend after my jobs done on auto-gptq

[BenjaminBossan]

That would be fantastic, big thanks.

[pacman100]

Thank you so much @PanQiWei for the impressive work on extending AdaptionPrompt to more models and adding the Multi-Modal AdaptionPromptV2 🔥🚀✨.

It would be great if we can have a Multi-Modal AdaptionPromptV2 example script or notebook in examples folder so that users can leverage it out of the box. Apart from that good to merge.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.