Question for Vulkan maintainers about gfx1103 / 780M (MMVQ related)

[arch-btw]

Hi guys,

Disclaimer: I'm kind of in over my head with this and used AI for the patch, but figured it may be worth mentioning.

I was reading this comment from @reuvenneumann here:

#20831 (comment)

So I got curious and looked into whether the same applied to the Vulkan backend, and wrote a small patch which gave me a +12.2% increase in tg speed. I'm wondering if this is something that's already known or worth looking into more? I just did a quick speed test with the same prompt, nothing more.

gemma-4-26B-A4B-it-QAT-Q4_0.gguf

Metric	Before	After	Change
TG throughput	20.59 t/s	23.11 t/s	+12.2%
TG latency	48.57 ms/tok	43.27 ms/tok	−10.9%

Patch:

#!/usr/bin/env python3
# patch_vk_narrow_mmvq.py
#
# Patches ggml/src/ggml-vulkan/ggml-vulkan.cpp to apply a conservative
# narrow-MoE MMVQ optimization for AMD RDNA3 iGPUs such as gfx1103.
#
# Usage:
#   python3 patch_vk_narrow_mmvq.py path/to/ggml-vulkan.cpp
#
# From llama.cpp repo root:
#   python3 patch_vk_narrow_mmvq.py ggml/src/ggml-vulkan/ggml-vulkan.cpp

import pathlib
import re
import shutil
import sys

MARKER = "GGML-VK RDNA3-UMA narrow MMVQ patch"
THRESHOLD = 2048
ARCH = "vk_device_architecture::AMD_RDNA3"


def die(msg: str) -> None:
    print(f"ERROR: {msg}", file=sys.stderr)
    sys.exit(1)


def patch_dmmv_wg_selectors(text: str) -> tuple[str, int]:
    """
    Patch:
      - ggml_vk_get_dequantize_mul_mat_vec()
      - ggml_vk_get_dequantize_mul_mat_vec_id()

    so that AMD RDNA3 iGPUs keep DMMV_WG_SIZE_SUBGROUP for narrow K.
    """
    old = "uint32_t dmmv_wg = DMMV_WG_SIZE_SUBGROUP;"

    funcs = [
        "static vk_pipeline ggml_vk_get_dequantize_mul_mat_vec(",
        "static vk_pipeline ggml_vk_get_dequantize_mul_mat_vec_id(",
    ]

    patched = 0

    for func_sig in funcs:
        start = text.find(func_sig)
        if start == -1:
            die(f"could not find function signature: {func_sig}...")

        idx = text.find(old, start)
        if idx == -1:
            die(f"could not find '{old}' inside: {func_sig}...")

        line_start = text.rfind("\n", 0, idx) + 1
        indent = text[line_start:idx]

        new = (
            old
            + "\n\n"
            + indent
            + f"// {MARKER}: keep one subgroup for narrow K.\n"
            + indent
            + "if (ctx->device->vendor_id == VK_VENDOR_ID_AMD &&\n"
            + indent
            + f"    ctx->device->architecture == {ARCH} &&\n"
            + indent
            + f"    ctx->device->uma && k <= {THRESHOLD}u) {{\n"
            + indent
            + "    dmmv_wg = DMMV_WG_SIZE_SUBGROUP;\n"
            + indent
            + "}"
        )

        text = text[:idx] + new + text[idx + len(old):]
        patched += 1

    return text, patched


def patch_should_use_mmvq(text: str) -> str:
    """
    Patch ggml_vk_should_use_mmvq().

    Current upstream AMD logic often rejects MMVQ when k < 2048.
    For RDNA3 iGPUs, one-subgroup MMVQ can be better for narrow MoE
    matrices during token generation, so allow it for n == 1.
    """
    pat = re.compile(
        r"(case\s+VK_VENDOR_ID_AMD:\s*\r?\n"
        r"[ \t]*if\s*\(\s*k\s*<\s*2048\s*\)\s*\{\s*\r?\n)"
        r"([ \t]*)"
        r"(return\s+false;)",
        re.M,
    )

    m = pat.search(text)
    if not m:
        die(
            "could not find the AMD 'if (k < 2048) { return false; }' block "
            "inside ggml_vk_should_use_mmvq()"
        )

    indent = m.group(2)

    replacement = (
        m.group(1)
        + indent
        + f"// {MARKER}: allow one-subgroup MMVQ for narrow MoE matrices on RDNA3 iGPU.\n"
        + indent
        + "if (n == 1 && device->architecture == "
        + ARCH
        + " && device->uma) {\n"
        + indent
        + "    return true;\n"
        + indent
        + "}\n"
        + indent
        + m.group(3)
    )

    return text[:m.start()] + replacement + text[m.end():]


def main() -> None:
    if len(sys.argv) > 1:
        path = pathlib.Path(sys.argv[1])
    else:
        path = pathlib.Path("ggml/src/ggml-vulkan/ggml-vulkan.cpp")

    if not path.is_file():
        die(
            f"file not found: {path}\n"
            "Run from the llama.cpp root directory or pass the file path explicitly."
        )

    text = path.read_text(encoding="utf-8")

    if MARKER in text:
        print("Already patched; nothing to do.")
        return

    backup = path.with_name(path.name + ".bak")
    if not backup.exists():
        shutil.copy2(path, backup)
        print(f"Backup written: {backup}")

    text, count = patch_dmmv_wg_selectors(text)
    if count != 2:
        die(f"expected to patch 2 MMVQ workgroup selectors, patched {count}")

    text = patch_should_use_mmvq(text)

    if MARKER not in text:
        die("patch failed unexpectedly")

    path.write_text(text, encoding="utf-8")

    print(f"Patched: {path}")
    print("")
    print("Next step: rebuild llama.cpp with Vulkan enabled.")

if __name__ == "__main__":
    main()

Thank you for your time.

[Miko997]

One thing stands out in the patch script: the workgroup change assigns DMMV_WG_SIZE_SUBGROUP immediately after initializing dmmv_wg to that same value. That inserted if doesn't change the value or prevent later code from overwriting it.

I'd test the ggml_vk_should_use_mmvq() change on its own first. That would help establish whether the speedup comes from selecting MMVQ rather than changing the workgroup size.

Could you share the resulting C++ diff and base commit? The final diff would be easier to review than the replacement script. For validation, I'd compare baseline, selector-only and full-patch builds using repeated identical benchmarks, then check numerical correctness for the affected operations.

Also, the condition covers AMD RDNA3 UMA devices generally, not just the 780M.

[arch-btw]

Thank you @Miko997 for taking the time to look at this! I think you're right about how you narrowed it down!

As for the baseline, when I tried the patch I believe it was on this commit: f172be7
However, to possibly make things easier, I just updated llama.cpp to the latest commit and ran the tests.

So that would be base commit: 4da6337

I've also made them into diff/patch files, you're right that's a lot easier, let me know if this wasn't what you meant as I'm not completely sure:

rdna3_mmvq_selector_only.patch

diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
index 414e14f85..8fab308e9 100644
--- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp
+++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
@@ -6556,6 +6556,10 @@ static bool ggml_vk_should_use_mmvq(const vk_device& device, uint32_t m, uint32_
         }
     case VK_VENDOR_ID_AMD:
         if (k < 2048) {
+            // Allow one-subgroup MMVQ for narrow MoE matrices on RDNA3 iGPUs (e.g. gfx1103)
+            if (n == 1 && device->architecture == vk_device_architecture::AMD_RDNA3 && device->uma) {
+                return true;
+            }
             return false;
         }
 

rdna3_narrow_mmvq.patch (this was the full original patch)

diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
index 414e14f85..17197a020 100644
--- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp
+++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
@@ -5551,6 +5551,13 @@ static vk_pipeline ggml_vk_get_dequantize_mul_mat_vec(ggml_backend_vk_context *
 
     // heuristic to choose workgroup size
     uint32_t dmmv_wg = DMMV_WG_SIZE_SUBGROUP;
+
+    // GGML-VK RDNA3-UMA narrow MMVQ patch: keep one subgroup for narrow K.
+    if (ctx->device->vendor_id == VK_VENDOR_ID_AMD &&
+        ctx->device->architecture == vk_device_architecture::AMD_RDNA3 &&
+        ctx->device->uma && k <= 2048u) {
+        dmmv_wg = DMMV_WG_SIZE_SUBGROUP;
+    }
     if ((ctx->device->vendor_id == VK_VENDOR_ID_NVIDIA && ctx->device->architecture != vk_device_architecture::NVIDIA_PRE_TURING) || ctx->device->vendor_id == VK_VENDOR_ID_INTEL) {
         // Prefer larger workgroups when M is small, to spread the work out more
         // and keep more SMs busy.
@@ -5639,6 +5646,13 @@ static vk_pipeline ggml_vk_get_dequantize_mul_mat_vec_id(ggml_backend_vk_context
 
     // heuristic to choose workgroup size
     uint32_t dmmv_wg = DMMV_WG_SIZE_SUBGROUP;
+
+    // GGML-VK RDNA3-UMA narrow MMVQ patch: keep one subgroup for narrow K.
+    if (ctx->device->vendor_id == VK_VENDOR_ID_AMD &&
+        ctx->device->architecture == vk_device_architecture::AMD_RDNA3 &&
+        ctx->device->uma && k <= 2048u) {
+        dmmv_wg = DMMV_WG_SIZE_SUBGROUP;
+    }
     if ((ctx->device->vendor_id == VK_VENDOR_ID_NVIDIA && ctx->device->architecture != vk_device_architecture::NVIDIA_PRE_TURING) || ctx->device->vendor_id == VK_VENDOR_ID_INTEL) {
         // Prefer larger workgroups when M is small, to spread the work out more
         // and keep more SMs busy.
@@ -6556,6 +6570,10 @@ static bool ggml_vk_should_use_mmvq(const vk_device& device, uint32_t m, uint32_
         }
     case VK_VENDOR_ID_AMD:
         if (k < 2048) {
+            // GGML-VK RDNA3-UMA narrow MMVQ patch: allow one-subgroup MMVQ for narrow MoE matrices on RDNA3 iGPU.
+            if (n == 1 && device->architecture == vk_device_architecture::AMD_RDNA3 && device->uma) {
+                return true;
+            }
             return false;
         }
 

Benchmarks

Baseline:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD open-source driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q4_0            |  13.43 GiB |    25.23 B | Vulkan     |  99 |           pp512 |        500.37 ± 1.83 |
| gemma4 26B.A4B Q4_0            |  13.43 GiB |    25.23 B | Vulkan     |  99 |           tg128 |         26.30 ± 0.02 |

build: 4da633776 (11223)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD open-source driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q4_0            |  13.43 GiB |    25.23 B | Vulkan     |  99 |           tg512 |         25.15 ± 0.17 |

build: 4da633776 (11223)

rdna3_mmvq_selector_only.patch

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD open-source driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q4_0            |  13.43 GiB |    25.23 B | Vulkan     |  99 |           pp512 |        498.52 ± 2.28 |
| gemma4 26B.A4B Q4_0            |  13.43 GiB |    25.23 B | Vulkan     |  99 |           tg128 |         27.49 ± 0.08 |

build: 4da633776 (11223)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD open-source driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q4_0            |  13.43 GiB |    25.23 B | Vulkan     |  99 |           tg512 |         26.02 ± 0.14 |

build: 4da633776 (11223)

rdna3_narrow_mmvq.patch

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD open-source driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q4_0            |  13.43 GiB |    25.23 B | Vulkan     |  99 |           pp512 |        501.65 ± 1.91 |
| gemma4 26B.A4B Q4_0            |  13.43 GiB |    25.23 B | Vulkan     |  99 |           tg128 |         27.54 ± 0.03 |

build: 4da633776 (11223)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD open-source driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q4_0            |  13.43 GiB |    25.23 B | Vulkan     |  99 |           tg512 |         26.15 ± 0.22 |

build: 4da633776 (11223)