Nomic Vulkan backend

[cebtenzzre]

This is Nomic's Kompute-based Vulkan backend from the GPT4All project, now available under the MIT license. It can be enabled by building with cmake and passing -DLLAMA_KOMPUTE=ON (make is currently not supported).

Structure

• The C++ code for the backend is in ggml-kompute.cpp and ggml-kompute.h. This is MIT licensed.
• Shaders are in a folder called kompute-shaders. These are based on the Metal backend and MIT licensed.
• Nomic's fork of Kompute is provided as a git submodule. This is Apache-2.0 licensed.

Limitations

• There is currently no partial offload support, so it is either -ngl 1 or -ngl 0, like Metal. We plan to implement this eventually, by implementing a split point in the compute graph. We do not plan to implement per-op offload (what most backends, do, including 0cc4am's Vulkan backend). (Partial offloading is now implemented.)
• Supported model architectures are currently Falcon and Llama.
• Supported model formats are Q4_0, Q4_1, FP16, and FP32, with or without Q6_K output tensors.
• GPU-accelerated matmul for prompt processing is currently disabled due to a known issue with incorrect output. Token generation runs 100% on the GPU. (GPU prompt processing has been fixed and re-enabled.)
• Discrete NVIDIA and AMD GPUs have been tested successfully, but both integrated and discrete Intel GPUs are currently known not to work (garbage output).
• This backend currently requires shaderFloat16, which e.g. not available in Pascal (GTX 10-series) and older GPUs from NVIDIA. You can check if your GPU is currently supported by going to vulkan.gpuinfo.org, finding your device, and searching for shaderFloat16 under "Features > Core 1.2" - it must be supported.
• Windows and Linux are fully supported. macOS has Metal so we haven't attempted to build this backend there.
• This PR still needs to be updated for the per-layer KV cache. Done a while ago.

Contributions are welcome! This backend currently works well enough for Nomic's purposes so at the moment we are not focused on adding new features, only maintenance - our team is quite small right now. The goal of this PR is to reduce the maintenance burden for us, as llama.cpp frequently introduces changes that affect all backends.

[slaren]

@ggerganov My concern is that a call to ggml_vk_init_device is required before the backend can be used. This creates two problems:

• Adds unnecessary backend-specific code to llama.cpp
• Forces other ggml applications that want to use this backend to add more backend-specific code

While I understand that this is not very high priority, the fix should be very simple (just move the initialization calls to the backend code), so I don't really see any reason to not fix it before merging.

[cebtenzzre]

It does not use GPU for the moment.

Right now only Falcon and Llama are whitelisted. This backend is strict about the model format because it is designed to run a contiguous graph on the GPU (like Metal), instead of only offloading the ops that are implemented (like Occam's Vulkan PR).

[0cc4m]

It does not use GPU for the moment.

Right now only Falcon and Llama are whitelisted. This backend is strict about the model format because it is designed to run a contiguous graph on the GPU (like Metal), instead of only offloading the ops that are implemented (like Occam's Vulkan PR).

At this point my backend also runs the full graph on GPU contiguously (if all layers are offloaded).

[rajeevn1]

ggml_vk_graph_compute: error: unsupported op 'MUL_MAT'

I get this error using kompute backend, on Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)

[cebtenzzre]

ggml_vk_graph_compute: error: unsupported op 'MUL_MAT'

What model are you using? Maybe the fallback code got broken among the recent changes (edit: yes, it did) - it sounds like you are using an unsupported quantization type.

Also, Intel GPUs are not currently known to work, and I don't know how well integrated GPUs work in general at the moment.

[rajeevn1]

I am using mistral-7b-instruct-v0.2.Q5_K_M.gguf

[Titaniumtown]

ggml_vk_graph_compute: error: unsupported op 'MUL_MAT'

I get this error using kompute backend, on Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)

I can reproduce on Alder Lake (Iris Xe)

[rajeevn1]

I tried with different quantization mistral-7b-instruct-v0.2.Q4_K_M.gguf but get the same error

ggml_vk_graph_compute: error: unsupported op 'MUL_MAT'

[cebtenzzre]

I tried with different quantization mistral-7b-instruct-v0.2.Q4_K_M.gguf but get the same error

This backend currently only supports Q4_0, Q4_1, and F16 quantizations. The latest master of llama.cpp will at least fall back to CPU in this case instead of failing.

[sorasoras]

I tried with different quantization mistral-7b-instruct-v0.2.Q4_K_M.gguf but get the same error

This backend currently only supports Q4_0, Q4_1, and F16 quantizations. The latest master of llama.cpp will at least fall back to CPU in this case instead of failing.

It would be nice if it can fallback to the other vulkan backend.
Just use two vulkan backend together lol.

[niansa]

I am really happy the Kompute implementation finally made it into mainline llama.cpp!

[akingoverlook]

This is Nomic's Kompute-based Vulkan backend from the GPT4All project, now available under the MIT license. It can be enabled by building with cmake and passing -DLLAMA_KOMPUTE=ON (make is currently not supported).

Structure

Nomic appears to claim support for Qualcomm GPUs by "Nomic Vulkan":
"September 18th, 2023: Nomic Vulkan launches supporting local LLM inference on AMD, Intel, Samsung, Qualcomm and NVIDIA GPUs."

Perhaps that is different from "Nomic's Kompute based Vulkan backend", that is not exactly clear. But what is clear is that this backend can't run on Qualcomm GPUs at all because it wants uniformAndStorageBuffer (8 and 16bit) access, which their Vulkan driver does not show as supported (on any of their GPUs).

FYI, it also can't run on ARM GPUs, because they have maximum subgroup size of 16 (even in the top of the line Immortalis chips), and this backend wants 32.

Any chance these dependencies could be worked around, @cebtenzzre ?