Eval bug: mmproj on vulkan extremely degraded compared to cuda for specific images

[Interpause]

Name and Version

>>> ./build/bin/llama-server --version
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (RADV PHOENIX) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = NVIDIA GeForce RTX 4090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
version: 8189 (4d828bd1a)
built with GNU 15.2.1 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

AMD Ryzen 7 PRO 8845HS (16) @ 5.14 GHz
NVIDIA GeForce RTX 4090 [Discrete]
AMD Radeon 780M Graphics [Integrated]

Models

https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/blob/main/Qwen3.5-27B-UD-Q3_K_XL.gguf
https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/blob/main/mmproj-F32.gguf

Problem description & steps to reproduce

For context, I use MTMD_BACKEND_DEVICE=Vulkan0 to toggle on/off whether mmproj-F32.gguf is offloaded to iGPU (Vulkan) or dGPU (CUDA). In both cases, Qwen3.5-27B-UD-Q3_K_XL.gguf is entirely offloaded to CUDA.

For most common cases like OCR, the vision performance is as expected. However, some images (this + 1 other I cannot publish) completely bamboozle the model's vision when on Vulkan versus CUDA:

describe this is one sentence.

CUDA's response:

The image shows a Gazebo Sim environment running in a browser via noVNC, displaying an orange quadcopter drone hovering above a gray platform amidst blue cloud-like obstacles, with a control panel on the right showing velocity settings and keyboard controls for forward, back, left, right, up, down, and yaw movements.

Vulkan's response:

The image displays a list of various words in Vietnamese, such as "tinh," "thanh," and "trường," arranged vertically against a plain white background.

I have also tested with mmproj-F16.gguf also from Unsloth. I ran into a similar issues with Qwen3.5-35B-A3B with both Unsloth's and AesSedai's model and mmproj, so I believe the issue is likely with the Vulkan implementation rather than the model.

EDIT: No issue with CPU/BLAS (MTMD_BACKEND_DEVICE=BLAS):

BLAS response:

This screenshot shows a Gazebo Sim simulation running in a browser via noVNC, displaying a drone flying among blue cloud-like obstacles with a control panel on the right for adjusting velocity and keyboard input mapping.

First Bad Commit

Unsure.

Relevant log output

cuda.log

vulkan.log

blas.log

[Interpause]

sorry might be dupe of #17438

[Interpause]

Let me see if #20589 fixes it

[Interpause]

not fixed unfortunately.

[0cc4m]

I can't reproduce this directly. Your image works for me in CUDA and in Vulkan:

This screenshot shows a Gazebo Sim simulation running in a web browser, displaying an orange drone flying through a sky filled with blue clouds while a control panel on the right allows the user to adjust velocity and yaw using keyboard inputs.

Also, your vulkan.log run used cuda.

[Interpause]

Also, your vulkan.log run used cuda.

To make it clear, the LLM is on CUDA, while the mmproj model is on Vulkan in the logs:

[52717] clip_ctx: CLIP using Vulkan0 backend
[52717] load_hparams: projector:          qwen3vl_merger
[52717] load_hparams: n_embd:             1152
[52717] load_hparams: n_head:             16
[52717] load_hparams: n_ff:               4304
[52717] load_hparams: n_layer:            27
[52717] load_hparams: ffn_op:             gelu
[52717] load_hparams: projection_dim:     5120
[52717] 
[52717] --- vision hparams ---
[52717] load_hparams: image_size:         768
[52717] load_hparams: patch_size:         16
[52717] load_hparams: has_llava_proj:     0
[52717] load_hparams: minicpmv_version:   0
[52717] load_hparams: n_merge:            2
[52717] load_hparams: n_wa_pattern: 0
[52717] load_hparams: image_min_pixels:   1048576 (custom value)
[52717] load_hparams: image_max_pixels:   4194304
[52717] 
[52717] load_hparams: model size:         1757.55 MiB
[52717] load_hparams: metadata size:      0.12 MiB
[52717] warmup: warmup with image size = 1472 x 1472
[52717] alloc_compute_meta:    Vulkan0 compute buffer size =   248.10 MiB
[52717] alloc_compute_meta:        CPU compute buffer size =    24.93 MiB
[52717] alloc_compute_meta: graph splits = 1, nodes = 823
[52717] warmup: flash attention is enabled
[52717] srv    load_model: loaded multimodal model, '/home/jh2xl/code/models/llm/Qwen3.5-27B_mmproj-F32.gguf'

All tests were conducted with LLM on CUDA with mmproj on the different backends. Maybe that could be a hint for what the problem is?

That said, I haven't tested in a while (I am currently using CUDA + RPC-ROCm), let me get back to you later if its still a problem.

[Interpause]

Still seemingly a problem:

vulkan-new.log

> build/bin/llama-server --version
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24082 MiB):
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, VRAM: 24082 MiB
version: 8981 (d77599234)
built with GNU 15.2.1 for Linux x86_64

> build/bin/llama-server --list-devices
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24082 MiB):
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, VRAM: 24082 MiB
Available devices:
  CUDA0: NVIDIA GeForce RTX 4090 (24082 MiB, 23687 MiB free)
  Vulkan0: AMD Radeon 780M Graphics (RADV PHOENIX) (65354 MiB, 65258 MiB free)
  Vulkan1: NVIDIA GeForce RTX 4090 (24564 MiB, 23687 MiB free)
  BLAS: BLIS (0 MiB, 0 MiB free)