Support More Models for Adaption Prompt and Add Adaption Prompt V2 with multi-modal ability supportion

[PanQiWei]

What does this pr do

This pr aims to add more models support for adaption prompt and adaption prompt v2 and make adaption prompt v2 support multi-modal training and inference

---

Should I directly extend it in adaption_prompt.py to support features of llama-adapter-v2 or create a new one named adaption_prompt_v2.py?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[PanQiWei]

From the official code of llama-adapter-v2 here, I think we can easily extend adaption_prompt.py to let users optionally choose new features such as bias tuning and layer-norm tuning to improve their models. What's your opinion @pacman100 ?

[PanQiWei]

this branch extend adaption_prompt_v2.py to support multi-modal training and fixed some bugs, once everything is stable and tests are built, I will close this pr and creat a new one that based on multi-modal branch.

[pacman100]

This is super! 🚀🔥✨

Thank you @PanQiWei for the incredible effort you have put wrt improving the AdaptionPrompt method and adding the AdaptionPromptV2 method! 🤗

LGTM! Maybe, in a separate PR we can add examples on using these methods?

[pacman100]

Could you please do make style and make quality as well as resolve the conflict so that we can merge this PR

[pacman100]

Hello, it would be great if you could add a test file for testing the minimal core components only (forward, save_pretrained, generate). cc @younesbelkada

[younesbelkada]

Thanks a lot for the great addition!
As @pacman100 suggested it would be great indeed if you can add a test file so that this addition will not be affected by future PRs
Please have a look at what has been done here for instance: https://github.com/huggingface/peft/blob/main/tests/test_adaption_prompt.py and you can adapt it to your needs. Let us know if you need help designing the tests.

[pacman100]

Hello @PanQiWei, please let us know if you need any help with this PR. Looking forward to merging it soon 🤗

[PanQiWei]

@pacman100 Hi, sorry for not replying for so long, I just so busy last month with other works and missed many messages in github. I will pickup some time to improve the pr(s), maybe separate adaption_prompt and adaption_prompt_v2 into two prs, I will done those before end of next week as soon as possible 🙏

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.