Misc. bug: 'llama-server --list-devices' doesn't show the CPU device

[yurivict]

Name and Version

$ llama-cli --version
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 2060 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
version: 6795 (unknown)
built with FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) for x86_64-unknown-freebsd14.3

Operating systems

BSD

Which llama.cpp modules do you know to be affected?

llama-server

Command line

$ llama-server --list-devices
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 2060 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
Available devices:
  Vulkan0: NVIDIA GeForce RTX 2060 (6144 MiB, 5136 MiB free)

Problem description & steps to reproduce

The CPU device isn't present in the list even though libggml-cpu.so is installed:

       /usr/local/lib/libggml-base.so
        /usr/local/lib/libggml-cpu.so
        /usr/local/lib/libggml-vulkan.so
        /usr/local/lib/libggml.so

First Bad Commit

n/a

Relevant log output

n/a

[skoulik]

As far as I can see, this is WAD. Has been this way for a while.

llama.cpp/common/arg.cpp

Line 2839 in 230d116

if (ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_CPU) {

llama.cpp/tools/llama-bench/llama-bench.cpp

Line 638 in 230d116

if (ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_CPU) {

llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp

Line 4900 in 230d116

if ((new_props.properties.deviceType == vk::PhysicalDeviceType::eDiscreteGpu || new_props.properties.deviceType == vk::PhysicalDeviceType::eIntegratedGpu) && ggml_vk_device_is_supported(devices[i])) {

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[yurivict]

@skoulik

It is bad that once ggml is compiled with Vulkan enabled llama can't allow to use the CPU backend at all.
llama-server fails on models that don't fit into the Vulkan memory. It doesn't attempt to place excess layers into CPU.

Is this fixable?

[0cc4m]

That is just not how it works. The CPU backend always sits in the background. You can run set num of gpu layers to 0 and it will mostly run on CPU, only offload some large operations. If you want true 100% CPU inference you need to use a CPU build or disable the GPU backend (e.g. by setting GGML_VK_VISIBLE_DEVICES= for Vulkan)