How do i specify which gpu to use for kv cache? How to offload expert tensors to specific gpu?

[milpster]

The title says it all. I was looking how i could offload expert tensors to a specific gpu. And i am looking to find a way to do the same with the kv cache.

Reason being is that i have a weak and a strong gpu and i want only the non expert tensors on the strong gpu, while putting everything else on the weaker gpu.

[woof-dog]

Take a look at the -ot option, aka --offload-tensors. It lets you select exactly where each tensor goes.

-ot,   --override-tensor <tensor name pattern>=<buffer type>,...
                                        override tensor buffer type
                                        (env: LLAMA_ARG_OVERRIDE_TENSOR)

You can do this like for example

-ot ".ffn_(up|down)_exps.=cuda1,.ffn_(gate)_exps.=cuda0"

Also look at --device

-dev,  --device <dev1,dev2,..>          comma-separated list of devices to use for offloading (none = don't
                                        offload)
                                        use --list-devices to see a list of available devices
                                        (env: LLAMA_ARG_DEVICE)

You might be able to mix the two, for example use --device to set the primary GPU and then -ot for overriding the expert tensors to the weaker GPU. Then also set -ngl 999 or however many of the expert tensors you want on the strong GPU.

I think where the kv cache goes is determined by --device, but I am not completely sure.

[zcattacz]

The old GPUs have distinct performance on fp32, fp16, int8, maybe if KV cache can be offloaded to a different card, one can leverage this to salvage the VRAM from old card like P40 24G with specific quant, which would be crawling slow with Q4 or fp16.