[Feature Request] Multimodality support (LLAVA)

[JianbangZ]

🚀 Feature

Multimodality model support (LLAVA)
There has been more and more community interest in the multimodality models, such as LLAVA. LLAVA itself has quite simple architecture: CLIP+projector+LLM.

[Kathryn-cat]

Hey @JianbangZ , thanks for bringing this up! We're bringing in LLaVA support this coming week after some major announcements are finished. We'll also bring a simple Python API that handles multimodality.

[brianjking]

Woah, why specifically LLaVa? Any plans for InstructBlip or other options?

Either way, VERY cool. @Kathryn-cat

[JianbangZ]

Woah, why specifically LLaVa? Any plans for InstructBlip or other options?

Either way, VERY cool. @Kathryn-cat

I think mainly due to it's simpleness. straight matmul projector instead of cross attention etc.

[JianbangZ]

@Kathryn-cat Great news! Will the entire pipeline be able to run on Vulkan or CUDA? perticularly the CLIP visual encoder part as LLM already has great Vulkan/CUDA support.

[saad-palapa]

Any updates on adding LLaVA to MLC?

[sam1am]

Really looking forward to this. Is it still happening?

[MasterJH5574]

Hello folks, llava has been supported in #1974 and other followup PRs recently (many thanks to @anibohara2000!!). You are more than welcome to try out, and are welcome to open new issues if there are errors or questions regarding the Llava support.