Hybrid CPU and GPU inference?

[billblake2018]

I'm using b4762, but I can upgrade at will. On my hardware and according to llama-bench, prompt generation is faster if I use the CPU and 8 threads, but token generation is faster if I use the (Intel Graphics via Vulkan) GPU and 1 thread. Moreover, the token generation speed decreases the more layers I offload to the GPU. Is there a reasonable way to make llama.cpp distribute its work so as to get the best of both worlds or must I stick with either CPU or GPU?

In case it is of interest: With CPU with 8 threads, pp is 21.65 and tg is 6.73. With GPU with 1 thread and 0 layers, it is 17.97 and 3.82. When I offload all layers to the GPU, it is 24.31 and 2.61. So using the GPU really kills token generation speed while at best providing only a modest improvement in prompt processing speed. This is with a llama 3.2 3B Q8 model on an HP ProBook with 32G running FreeBSD 14.1.

[ejrydhfs]

This might be of interest #4543 it's possible to efficiently distribute work aka shard word between CPU and GPU and achieve tensor parallelization between the two aka have the two work at the exact same time, but it's experimental. There's more info here https://github.com/SJTU-IPADS/PowerInfer

[alosslessdev]

also, using the -ngl argument on llama-server seems to have a sweet spot if you use it with GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 and compile llama.cpp with cuda to maximize token generation. you load most layers in memory like 38 if you use a model that has 40, using -ngl 38. the numbers of layers you load depends on the amount of vram you have since they are loaded in vram by default if you compile with cuda. since i have 6 gb of vram i loaded 38 layers. if you load more, token speed falls off a cliff, and if you load less, you leave speed on the table. if you don't use unified memory and you use -ngl only you leave performance on the table, and if you don't use -ngl and use only unified memory it will load all layers reducing performance. performance is maximized by making sure the pcie link between gpu and cpu is not saturated, which can become a bottleneck if you load more layers into the gpu using unified memory

i am running the following command on 6gb of vram on an rtx 3060 mobile, with a 5600h cpu and pcie 3.0 x8 connection to gpu: GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ./llama-server -hf ggml-org/GLM-4.6V-Flash-GGUF -ngl 38 -cram 0

i do not benefit from kv cache so i disabled it with -cram 0, -ngl is what enables hybrid inference

i am pretry sure this is the practical effect powerinfer has, by not saturating the pcie link. maybe it's not necessary anymore

since i am using a vision model, token generation speed varies a lot depending on image content and speed may fall off a cliff depending on the image. so you will have to reduce the number of layers. maybe its the same for other multimodal models and i guess it depends on the amount of tokens. the cliff can be at any position such as after 37, 38 layers etc and the maximum speed may be 20, 25 tok/s etc. the cliff is when you reach less than 1 tok/s

you may want to use binary search to find the optimal number of layers to load with -ngl

since hybrid inference is sequential aka communication between cpu and gpu here is synchronous, meaning one has to wait for the other to send data to continue, utilization never goes past 50% for either the cpu or the gpu. it is not possible to move more load to the gpu due to pcie bandwidth limitations, attempting to do so with a higher number of layers will lead to the cliff. and if you move more to the cpu by reducing -ngl the load on the gpu will be reduced. but this is the way i extracted the most performance out of my machine for a large model that doesn't fit entirely in vram

Without unified memory I can only offload 10 layers for vision