multimodal adapter

[TheShy-Dream]

I want to customize an adapter myself so that it can incorporate some multimodal information. Can I use this library。If possible, could you provide an example?

[PanQiWei]

Hi, I'm also interested in multi-modal LLMs' fine tuning, and thus I create this pr, but I'm doing with the algorithm proposed in llama-adapter-v2, how about yours?

[TheShy-Dream]

Hi,In fact, I also followed the idea of llama-adapter V2. I don't know if using peft can provide a fast implementation method.

[PanQiWei]

Sure, for in peft there is already have a adaption_prompt that is actually llama-adapter, and I've extended it to support more other models such as gptj and gpt-neox, I also already implemented adaption_prompt_v2 which is llama-adapter-v2 and extend to support other models and multi-modal information integration in training and inference

[TheShy-Dream]

Hi,Thank you for your reply. Can you provide an example about it, because I couldn't find any relevant examples in the examples or readme.md

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.