You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thank you for your interest in the OmniFusion model!
At the moment we do not have a 4bit version of the model, but we plan to publish a light version of the model based on 1B or 3B LLM. We will also think about 8bit/4bit versions of the model, thanks for the ideas and suggestions!
If you have multiple GPUs, you can try using 'auto' for the Mistral language model in the example code from readme. The remaining parts (adapter and visual encoder) do not use a lot of video memory and can be placed on one video card.
Hello, I'm really interested in your project. But it takes much vram to run. Could you please add inference code in 4bits using bitsandbytes or smth
The text was updated successfully, but these errors were encountered: