How best to use these Vulkan resources with llama.cpp?

[PilotBob42]

Recently made the switch from Ollama to llama.cpp (b10221) in vulkan mode and never looking back. But I wanted to ask the community how best to formulate the llama-server command to make the most of what I have.

I have dual Radeon Pro V620s and a Ryzen 7 5700G APU. I'm running models like gpt-oss and nemotron-3-super. Obviously, I would like the V620s to do the bulk of the heavy lifting and for any layers that don't fit in the discrete GPUs to overflow to the integrated GPU and/or CPU.

Is there anything more I should be doing (or anything I'm doing wrong) to get the most out of my setup?

This is my current command line to start llama.cpp (I use Open WebUI as my interface):

./llama-server --port 8088 --ctx-size 262144 --no-ui --sleep-idle-seconds 600 --models-max 1 --models-autoload -fa on -lm none -lv 1

I also set:

GGML_VK_VISIBLE_DEVICES=0,1,2

llama-server reports:

./llama-server --list-devices
Available devices:
Vulkan0: AMD Radeon Graphics (RADV RENOIR) (98330 MiB, 97438 MiB free)
Vulkan1: AMD Radeon Pro V620 (RADV NAVI21) (30704 MiB, 30669 MiB free)
Vulkan2: AMD Radeon Pro V620 (RADV NAVI21) (30704 MiB, 30669 MiB free)

[doug923]

For your specific question, it is not specifically related to Vulkan. For me to learn how to setup the llama-server, I mainly refer to

https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

and, ask the ai models to explain a particular terminology.

Regarding to your example, you might want to try.

./llama-server --port 8088 --ctx-size 262144 --no-ui --sleep-idle-seconds 600 --models-max 1 --models-autoload -fa on -lm none -lv 4 -v --device Vulkan1,Vulkan2,Vulkan0 -m /path/to/your-model

With -lv 4 flag, the log entries will show you the tensors and kv cache from with which layers were assigned to which gpu. If it does work as what you expect, you can remove -v and switch back to -lv 1.