How to use --mmproj

[SoftologyPro]

What model do I need to use for --mmproj so the ability to add images works?
I downloaded this one
https://huggingface.co/unsloth/InternVL3-2B-GGUF/resolve/main/mmproj-BF16.gguf
did seem to load when I passed the gguf to --mmproj, but the image option is still disabled with the label "Image processing requires a vision model"

I tried that mmproj model with gemma3, gemma4, gpt-oss and qwen3.6 and they all do not allow images.

Which vision model is needed to enable the + image option? Can someone point me to a gguf URL that works? Uncensored if possible.

Thanks.

[wqerrewetw]

Try FP16 if your GPU does not support BF16.

[SoftologyPro]

OK, I downloaded and tried the F16 version, ie

llama-server -m models\gemma4-26b.gguf --mmproj models\mmproj-F16.gguf --port 8080

and it gives this error

mtmd_init_from_file: error: mismatch between text model (n_embd = 2816) and mmproj (n_embd = 1536)
hint: you may be using wrong mmproj
srv    load_model: failed to load multimodal model, 'models\mmproj-F16.gguf'
srv    operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

Error when using the BF16 version

mtmd_init_from_file: error: mismatch between text model (n_embd = 2560) and mmproj (n_embd = 1536)
hint: you may be using wrong mmproj
srv    load_model: failed to load multimodal model, 'models\mmproj-BF16.gguf'
srv    operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

Without the --mmproj argument llama.cpp loads any other model and starts fine.

[shinishiho]

so which mmproj are you using? if you are using gemma4-26b-a4b, you might want to use the mmproj in the same repo, like https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/blob/main/mmproj-BF16.gguf
edit: not "same repo", but for the same type of model. for example, I use BeaverAI/Artemis-31b-v1e, which is a gemma4-31b tune, I can use the mmproj file in unsloth's gemma4 31b repo.

[SoftologyPro]

OK, that works with gemma4. So do I need to have a specific mmproj gguf matching each model I use?
And I get an out of memory crash when asking about even the smallest resolution image, so no good for a 24 GB GPU.

[shinishiho]

i believe so.
also, from https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md, might help organizing things a bit:

If the model contains multiple GGUF (for multimodal or multi-shard), files should be put into a subdirectory. The directory structure should look like this:

models_directory
 │
 │  # single file
 ├─ llama-3.2-1b-Q4_K_M.gguf
 ├─ Qwen3-8B-Q4_K_M.gguf
 │
 │  # multimodal
 ├─ gemma-3-4b-it-Q8_0
 │    ├─ gemma-3-4b-it-Q8_0.gguf
 │    └─ mmproj-F16.gguf   # file name must start with "mmproj"
 │
 │  # multi-shard
 ├─ Kimi-K2-Thinking-UD-IQ1_S
 │    ├─ Kimi-K2-Thinking-UD-IQ1_S-00001-of-00006.gguf
 │    ├─ Kimi-K2-Thinking-UD-IQ1_S-00002-of-00006.gguf
 │    ├─ ...
 │    └─ Kimi-K2-Thinking-UD-IQ1_S-00006-of-00006.gguf

There's a flag for llama-server, not sure if others have it:
--mmproj-offload, --no-mmproj-offload whether to enable GPU offloading for multimodal projector (default: enabled)
(env: LLAMA_ARG_MMPROJ_OFFLOAD)

I use the mentioned 31b model in a 24gb gpu, with kv cache quantization q8_0, 32768 ctx size, and that works
/app/llama-server --model /root/.cache/llama.cpp/Artemis-31B-v1e-Q4_K_M.gguf --mmproj /root/.cache/llama.cpp/mmproj-BF16.gguf --n-gpu-layers 9999 --host 0.0.0.0 --port 8080 --ctx-size 32768 --n-predict -1 --parallel 1 --flash-attn on --cache-ram 0 --sleep-idle-seconds 1 --cache-type-k q8_0 --cache-type-v q8_0 --no-webui --no-slots

[SoftologyPro]

OK, thank you for the explanations.

[shinishiho]

hope that helped ya

[SoftologyPro]

It did. Thank you.