Feature Request: Split model over multiple Vulkan GPUs

[wittypastoral]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Related to #5259 (closed), if you want I could move this there.

How hard would it be to implement splitting over vulkan GPUs instead of CUDA/HIP?

I guess OpenCL could be another path if vulkan is too hard, since there's now a maturing rusticl driver that can be layered on top of vulkan as well as various native drivers, but it may not be fully baked enough yet to support llamacpp (though maybe that's changing [1]). Also, afaik mapping memory between GPUs in a multi-GPU config is still under active development/implementation.

[1] https://archive.fosdem.org/2024/events/attachments/fosdem-2024-3364-why-not-run-opencl-accelerated-llm-on-your-phone-/slides/22383/Why_not_run_OpenCL-accelerated_LLM_on_your_phon_nK2DudB.pdf

Motivation

This would be really helpful as it's now not unreasonable to want to ditch nvidia's drivers for the open source NVK vulkan driver, and AMD's cards are also MUCH better supported with vulkan on the RADV driver than with AMD's spotty/nonexistant ROCm/HIP support. Vulkan is also more universally supported, so this could enable someone to split a model over eg. an AMD and an nvidia GPU if that's what they have.

Possible Implementation

N/A

[0cc4m]

I implemented Vulkan multigpu in #5321, it's been around for a while. It should work reasonably well, although there's probably optimizations left that could reduce device-to-device communication overhead.

[wittypastoral]

I implemented Vulkan multigpu in #5321, it's been around for a while. It should work reasonably well, although there's probably optimizations left that could reduce device-to-device communication overhead.

Oh damn, wow! Thanks!

[wittypastoral]

I implemented Vulkan multigpu in #5321, it's been around for a while. It should work reasonably well, although there's probably optimizations left that could reduce device-to-device communication overhead.

To be honest, I wouldn't be surprised if copying to ram instead of directly from device to device is also doing a good job of working around various issues with vulkan driver and memory sharing implementations.

[cb88]

I implemented Vulkan multigpu in #5321, it's been around for a while. It should work reasonably well, although there's probably optimizations left that could reduce device-to-device communication overhead.

To be honest, I wouldn't be surprised if copying to ram instead of directly from device to device is also doing a good job of working around various issues with vulkan driver and memory sharing implementations.

Yes but it also seems to prevent some performance scaling you'd expect with mgpu, eg some models I load on vulkan mgpu do not perform faster for prompt processing or text generation as you would expect perhaps but they do load balance ram at least.

But yes the one method we talked about yesterday would probably require GPUs on the same driver at a minimum and potentially identical GPUs.

[wittypastoral]

I implemented Vulkan multigpu in #5321, it's been around for a while. It should work reasonably well, although there's probably optimizations left that could reduce device-to-device communication overhead.

To be honest, I wouldn't be surprised if copying to ram instead of directly from device to device is also doing a good job of working around various issues with vulkan driver and memory sharing implementations.

Yes but it also seems to prevent some performance scaling you'd expect with mgpu, eg some models I load on vulkan mgpu do not perform faster for prompt processing or text generation as you would expect perhaps but they do load balance ram at least.

Do you mean not being faster than running on the CPU, or on a single GPU?

[cb88]

Do you mean not being faster than running on the CPU, or on a single GPU?

Roughly same speed as 1 GPU for me on 2x MI60 radv YMMV on other GPUs / drivers.

[wittypastoral]

Do you mean not being faster than running on the CPU, or on a single GPU?

Roughly same speed as 1 GPU for me on 2x MI60 radv YMMV on other GPUs / drivers.

Is there a serial dependency between the contents of each GPU in the current configuration? ie. needing to complete layers 1 2 and 3 on gpu1 before handing on the result to layers 4 5 and 6 on gpu2, or is it instead split more down the middle?

[0cc4m]

It's a regular layer split implementation, you split the model at some point and put half the layers on the first GPU and half the layers on the second GPU. Once the first GPU is done with its part the intermediate result gets copied to the second GPU and that one continues. It's the same way it works on CUDA and ROCm by default.

[wittypastoral]

ah, I guess running at the same speed would be to be expected, since there isn't any parallelization between GPUs.

[akumaburn]

@0cc4m How do we specify the device order for mixed-gpu setups with the Vulkan backend?

[0cc4m]

@0cc4m How do we specify the device order for mixed-gpu setups with the Vulkan backend?

Similar to CUDA, with GGML_VK_VISIBLE_DEVICES

[akumaburn]

@0cc4m How do we specify the device order for mixed-gpu setups with the Vulkan backend?

Similar to CUDA, with GGML_VK_VISIBLE_DEVICES

I'm running a test with two mismatched GPUs (a 7900xtx and a GT1030) and I seem to be running out memory on the 2nd device for some reason, while the first still has free vram.

load_tensors:      Vulkan0 model buffer size = 16708.36 MiB
load_tensors:      Vulkan1 model buffer size =  1799.99 MiB
load_tensors:   CPU_Mapped model buffer size =   417.66 MiB
llama_init_from_model: n_seq_max     = 1
llama_init_from_model: n_ctx         = 4096
llama_init_from_model: n_ctx_per_seq = 4096
llama_init_from_model: n_batch       = 512
llama_init_from_model: n_ubatch      = 512
llama_init_from_model: flash_attn    = 0
llama_init_from_model: freq_base     = 1000000.0
llama_init_from_model: freq_scale    = 1
llama_init_from_model: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 64, can_shift = 1
llama_kv_cache_init:    Vulkan0 KV buffer size =   960.00 MiB
llama_kv_cache_init:    Vulkan1 KV buffer size =    64.00 MiB
llama_init_from_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
llama_init_from_model: Vulkan_Host  output buffer size =     0.58 MiB
ggml_vulkan: Device memory allocation of size 385880064 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
ggml_gallocr_reserve_n: failed to allocate Vulkan1 buffer of size 385880064
ggml_vulkan: Device memory allocation of size 385880064 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
ggml_gallocr_reserve_n: failed to allocate Vulkan1 buffer of size 385880064
llama_init_from_model: failed to allocate compute buffers
common_init_from_params: failed to create context with model '/home/aeslampanah/Desktop/Projects/LLaMA/ultiima-32B.Q4_K_M.gguf'
main: error: unable to load model

Is this because of the context? Or something else? Because if it is the context, why not allocate accordingly to begin with?

[0cc4m]

You might have to use --tensor-split to finetune how much goes onto the second GPU. It's always more difficult if the VRAM capacities are very different.