Eval bug: Slowdown when using Vulkan Multi-GPU

[AbdullahMPrograms]

Name and Version

./llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
Device 0: AMD Radeon PRO W7900, gfx1100 (0x1100), VMM: no, Wave Size: 32
Device 1: AMD Radeon PRO W7900, gfx1100 (0x1100), VMM: no, Wave Size: 32
version: 6835 (5cca254)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

Vulkan

Hardware

2x Radeon Pro W7900

Models

unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF
unsloth/GLM-4.5-Air-GGUF

Problem description & steps to reproduce

When using Vulkan with split mode Row or Layer (Multi GPU), performance is significantly worse than split mode None (single GPU).

On the exact same configuration, ROCm does not see a massive performance degradation from split mode None to Layer.

Vulkan is significantly faster than ROCm for models that can fit in a single model due to this, but for models that require multiple GPU's it falls behind significantly. Ideally, the "None" split mode performance would be much closer to the layer performance as it is in ROCm, so bigger models can run faster

First Bad Commit

No response

Relevant log output

ultimis@ultimis-desktop:~$ ./LLM/llama.cpp/vulkan/bin/llama-bench -m /home/ultimis/LLM/Models/Qwen3-30B-A3B-Instruct-2507-UD-Q8_K_XL.gguf -ngl 999 -fa on -p 4096 -sm none,row,layer
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon PRO W7900 (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon PRO W7900 (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |    sm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | Vulkan     | 999 |  none |          pp4096 |       1481.89 ± 3.87 |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | Vulkan     | 999 |  none |           tg128 |        120.28 ± 0.07 |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | Vulkan     | 999 |   row |          pp4096 |       1347.79 ± 2.61 |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | Vulkan     | 999 |   row |           tg128 |         79.87 ± 0.10 |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | Vulkan     | 999 | layer |          pp4096 |       1343.55 ± 2.32 |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | Vulkan     | 999 | layer |           tg128 |         79.98 ± 0.09 |

build: cec5edbca (6798)

ultimis@ultimis-desktop:~$ ./LLM/llama.cpp/rocm/bin/llama-bench -m /home/ultimis/LLM/Models/Qwen3-30B-A3B-Instruct-2507-UD-Q8_K_XL.gguf -ngl 999 -fa on -p 4096 -sm none,row,layer
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon PRO W7900, gfx1100 (0x1100), VMM: no, Wave Size: 32
  Device 1: AMD Radeon PRO W7900, gfx1100 (0x1100), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl |    sm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | ROCm       | 999 |  none |          pp4096 |       1561.62 ± 4.64 |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | ROCm       | 999 |  none |           tg128 |         85.69 ± 0.14 |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | ROCm       | 999 |   row |          pp4096 |       1299.90 ± 6.33 |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | ROCm       | 999 |   row |           tg128 |         61.03 ± 0.06 |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | ROCm       | 999 | layer |          pp4096 |      1554.36 ± 12.90 |
| qwen3moe 30B.A3B Q8_0          |  33.51 GiB |    30.53 B | ROCm       | 999 | layer |           tg128 |         82.55 ± 0.21 |

build: cec5edbca (6798)


ultimis@ultimis-desktop:~$ ./LLM/llama.cpp/rocm/bin/llama-bench -m /home/ultimis/LLM/Models/unsloth/GLM-4.5-Air-GGUF/GLM-4.5-Air-Q2_K.gguf -ngl 999 -fa on -p 2048 -sm none,row,layer
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon PRO W7900, gfx1100 (0x1100), VMM: no, Wave Size: 32
  Device 1: AMD Radeon PRO W7900, gfx1100 (0x1100), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl |    sm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | --------------: | -------------------: |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | ROCm       | 999 |  none |          pp2048 |        195.31 ± 2.03 |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | ROCm       | 999 |  none |           tg128 |         51.46 ± 0.01 |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | ROCm       | 999 |   row |          pp2048 |        198.01 ± 0.52 |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | ROCm       | 999 |   row |           tg128 |         38.00 ± 0.04 |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | ROCm       | 999 | layer |          pp2048 |        299.73 ± 0.61 |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | ROCm       | 999 | layer |           tg128 |         52.06 ± 0.03 |

build: cec5edbca (6798)

ultimis@ultimis-desktop:~$ ./LLM/llama.cpp/vulkan/bin/llama-bench -m /home/ultimis/LLM/Models/unsloth/GLM-4.5-Air-GGUF/GLM-4.5-Air-Q2_K.gguf -ngl 999 -fa on -p 2048 -sm none,row,layer
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon PRO W7900 (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon PRO W7900 (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |    sm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | --------------: | -------------------: |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | Vulkan     | 999 |  none |          pp2048 |        464.58 ± 2.47 |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | Vulkan     | 999 |  none |           tg128 |         73.87 ± 0.17 |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | Vulkan     | 999 |   row |          pp2048 |        443.92 ± 0.78 |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | Vulkan     | 999 |   row |           tg128 |         43.49 ± 0.04 |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | Vulkan     | 999 | layer |          pp2048 |        442.43 ± 0.86 |
| glm4moe 106B.A12B Q2_K - Medium |  41.96 GiB |   110.47 B | Vulkan     | 999 | layer |           tg128 |         43.50 ± 0.04 |

build: cec5edbca (6798)

[jeffbolznv]

Split row isn't supported yet for vulkan. Split layer should work, but won't be faster unless the model doesn't fit in one GPU.

[AbdullahMPrograms]

Split row isn't supported yet for vulkan. Split layer should work, but won't be faster unless the model doesn't fit in one GPU.

Just to confirm, you mean the results I am getting with row and layer split are expected?

[jeffbolznv]

I'm surprised there's that much of a slowdown. I'd expect some overhead but the tg is a lot slower. I haven't spent much time using these split modes so I don't have a guess to the cause.

[AbdullahMPrograms]

I'm surprised there's that much of a slowdown. I'd expect some overhead but the tg is a lot slower. I haven't spent much time using these split modes so I don't have a guess to the cause.

I was quite shocked to see the slowdown too! Vulkan is significantly more performant than ROCm, but being limited to models that can only fit in one GPU is quite the compromise

[jeffbolznv]

I have asymmetric GPUs in my system, but the results I get match my expectations:

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>set GGML_VK_VISIBLE_DEVICES=1

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -p 0 -n 128 -fa 1 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |  1 |           tg128 |        281.93 ± 8.71 |

build: 55945d2ef (6834)

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -p 0 -n 128 -fa 1 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>set GGML_VK_VISIBLE_DEVICES=0

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -p 0 -n 128 -fa 1 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4070 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |  1 |           tg128 |        160.56 ± 6.97 |

build: 55945d2ef (6834)

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>set GGML_VK_VISIBLE_DEVICES=

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -p 0 -n 128 -fa 1 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -sm none,row,layer
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4070 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
| model                          |       size |     params | backend    | ngl |    sm | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |  none |  1 |           tg128 |       158.10 ± 10.62 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |   row |  1 |           tg128 |       202.89 ± 30.67 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 | layer |  1 |           tg128 |       204.26 ± 17.12 |

build: 55945d2ef (6834)

The perf I get for row/layer is between the perf of the two GPUs, and very close to what I'd estimate (1 / (0.5 / 280 + 0.5 / 160) == 203).

[netrunnereve]

Have you tried using a non MOE model? From my experience split mode works pretty well with RADV and it's only a bit slower than running on one card.

Also yeah you should be only using split mode layer with Vulkan, not row.

[AbdullahMPrograms]

I have asymmetric GPUs in my system, but the results I get match my expectations:

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>set GGML_VK_VISIBLE_DEVICES=1

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -p 0 -n 128 -fa 1 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |  1 |           tg128 |        281.93 ± 8.71 |

build: 55945d2ef (6834)

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -p 0 -n 128 -fa 1 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>set GGML_VK_VISIBLE_DEVICES=0

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -p 0 -n 128 -fa 1 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4070 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |  1 |           tg128 |        160.56 ± 6.97 |

build: 55945d2ef (6834)

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>set GGML_VK_VISIBLE_DEVICES=

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -p 0 -n 128 -fa 1 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -sm none,row,layer
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4070 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
| model                          |       size |     params | backend    | ngl |    sm | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |  none |  1 |           tg128 |       158.10 ± 10.62 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |   row |  1 |           tg128 |       202.89 ± 30.67 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 | layer |  1 |           tg128 |       204.26 ± 17.12 |

build: 55945d2ef (6834)

The perf I get for row/layer is between the perf of the two GPUs, and very close to what I'd estimate (1 / (0.5 / 280 + 0.5 / 160) == 203).

Thank you for the data point!

This is what I'd expect too! The performance of the layer split makes sense considering the GPU's individual performance for the model!

[AbdullahMPrograms]

Have you tried using a non MOE model? From my experience split mode works pretty well with RADV and it's only a bit slower than running on one card.

Also yeah you should be only using split mode layer with Vulkan, not row.

Similar result with Qwen 3-32B:

ultimis@ultimis-desktop:~$ ./LLM/llama.cpp/vulkan/bin/llama-bench -m /home/ultimis/LLM/Models/unsloth/Qwen3-32B-GGUF/Qwen3-32B-UD-Q4_K_XL.gguf -ngl 999 -fa on -p 2048 -sm none,row,layer
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon PRO W7900 (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon PRO W7900 (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |    sm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | --------------: | -------------------: |
| qwen3 32B Q4_K - Medium        |  18.64 GiB |    32.76 B | Vulkan     | 999 |  none |          pp2048 |        309.92 ± 2.55 |
| qwen3 32B Q4_K - Medium        |  18.64 GiB |    32.76 B | Vulkan     | 999 |  none |           tg128 |         31.67 ± 0.06 |
| qwen3 32B Q4_K - Medium        |  18.64 GiB |    32.76 B | Vulkan     | 999 |   row |          pp2048 |        300.59 ± 0.50 |
| qwen3 32B Q4_K - Medium        |  18.64 GiB |    32.76 B | Vulkan     | 999 |   row |           tg128 |         25.79 ± 0.03 |
| qwen3 32B Q4_K - Medium        |  18.64 GiB |    32.76 B | Vulkan     | 999 | layer |          pp2048 |        300.46 ± 0.37 |
| qwen3 32B Q4_K - Medium        |  18.64 GiB |    32.76 B | Vulkan     | 999 | layer |           tg128 |         25.71 ± 0.02 |

build: 5cca2542a (6835)

[jeffbolznv]

I get a bigger drop than I'd expect when disabling host-visible vidmem:

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -p 0 -n 128 -fa 1 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -sm none,row,layer
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4070 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
| model                          |       size |     params | backend    | ngl |    sm | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |  none |  1 |           tg128 |        158.92 ± 3.61 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |   row |  1 |           tg128 |       170.29 ± 22.97 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 | layer |  1 |           tg128 |       168.41 ± 24.71 |

Maybe there's some slow path in the tensor copies, or maybe the overhead from the copies is just really bad.

[AbdullahMPrograms]

I get a bigger drop than I'd expect when disabling host-visible vidmem:

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -p 0 -n 128 -fa 1 -r 10 --prio 1 -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -sm none,row,layer
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4070 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
| model                          |       size |     params | backend    | ngl |    sm | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |  none |  1 |           tg128 |        158.92 ± 3.61 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 |   row |  1 |           tg128 |       170.29 ± 22.97 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  99 | layer |  1 |           tg128 |       168.41 ± 24.71 |

Maybe there's some slow path in the tensor copies, or maybe the overhead from the copies is just really bad.

Are there any tests you'd like me to run on my side to help narrow down the issue?

[jeffbolznv]

I don't have any great ideas. One thing you could try would be to run with the env var GGML_VK_PERF_LOGGER=1 set and see if any of the operations is running slower. But I suspect the issue is with copying tensors between devices and that won't show up in the perf logger.

[AbdullahMPrograms]

I don't have any great ideas. One thing you could try would be to run with the env var GGML_VK_PERF_LOGGER=1 set and see if any of the operations is running slower. But I suspect the issue is with copying tensors between devices and that won't show up in the perf logger.

Incredibly long output with GGML_VK_PERF_LOGGER=1 here:
https://gist.github.com/AbdullahMPrograms/5f805dffaa4951f335ee899d4a1b1948

[jeffbolznv]

I think there's a lot of overhead from copying the tensors between devices. To make this faster we'd probably need to batch the copies.