ggml_vulkan: No devices found using Vulkan docker image.

[git2212]

My starting assumption is that I missed something although I tried following the instructions for compiling Vulkan support for a docker image at https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#for-docker-users.

Image compiles and the container runs using:

docker run -d --name llama-cpp-tmp -p 8010:8080 -v "/mnt/ai-models-llama:/models" --rm -ti --device /dev/dri/renderD128:/dev/dri/renderD128 --device /dev/dri/renderD129:/dev/dri/renderD129 llama-cpp-vulkan:b6503 ...
... but it loads everything onto the CPU...

running the command inside the docker container llama-server --list-devices returns

ggml_vulkan: No devices found.
load_backend: loaded Vulkan backend from /app/libggml-vulkan.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
Available devices:

rather odd considering that on the host vulkaninfo reports multiple gpus and ls -la /dev/dri:

crw-rw---- 1 nobody nogroup 226, 128 Sep 20 00:37 renderD128
crw-rw---- 1 nobody nogroup 226, 129 Sep 20 00:37 renderD129
crw-rw---- 1 nobody nogroup 226, 130 Sep 20 00:37 renderD130
crw-rw---- 1 nobody nogroup 226, 131 Sep 20 00:37 renderD131
crw-rw---- 1 nobody nogroup 226, 132 Sep 20 00:37 renderD132
crw-rw---- 1 nobody nogroup 226, 133 Sep 20 00:37 renderD133

ls -la /dev/dri inside the docker container reports:

crw-rw---- 1 nobody nogroup 226, 128 Sep 20 00:37 renderD128
crw-rw---- 1 nobody nogroup 226, 129 Sep 20 00:37 renderD129

which matches the two devices the container was started with...

The host has libvulkan1, mesa-vulkan-drivers, vulkan-tools, mesa-utils installed, as well as cuda-toolkit for docker... this host works fine with CUDA based llama.cpp container... ?!?

the host's uname -a :
Linux gpu-lxc-vllm-1 6.8.12-15-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-15 (2025-09-12T11:02Z) x86_64 GNU/Linux

Any suggestions are welcome.

[EnderHane]

I meet this problem and find a solution.

1. Create a Dockerfile with:

FROM ghcr.io/ggml-org/llama.cpp:server-vulkan

RUN apt-get update && apt-get install -y --no-install-recommends libglvnd0 libgl1 libglx0 libegl1 libgles2 && rm -rf /var/lib/apt/lists/*

(saved as llama.cpp-vulkan-nvidia.ContainerFile)

2. Build it:

podman build -t localhost/llama.cpp:server-vulkan -f llama.cpp-vulkan-nvidia.ContainerFile .

3. Run:

podman run -it --rm --gpus all localhost/llama.cpp:server-vulkan --list-devices

outputs

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4060 Ti (NVIDIA) | (...)

Story:

When llama.cpp-vulkan (either built by myself or pulled from ghcr.io) couldn't find my 4060Ti, docker.io/nvidia/vulkan:1.3-470 can find the GPU. The solution is to install some of the packages present in nvidia/vulkan but absent from llama.cpp-vulkan.

[VVakko]

I have slightly expanded the topic of refining the official full-vulkan image.
The result is a single compose.yml file that enables the use of not only NVIDIA graphics cards with the Vulkan backend but also the simultaneous use of NVIDIA and AMD cards.
Additionally, llama-server now runs inside the container with regular user privileges instead of root.

Note: The libnvidia-gl-580 package is installed because version 580 of the NVIDIA drivers is installed on the host; it will not work without this package.

services:
  llama-cpp:
    image: "localhost/llama.cpp:full-vulkan"
    container_name: "llama-cpp"
    build:
      ### Quickly add missing packages to an existing llama.cpp
      ### image for working with nVidia GPUs via Vulkan backend
      context: .
      dockerfile_inline: |
        FROM ghcr.io/ggml-org/llama.cpp:full-vulkan
        RUN  export DEBIAN_FRONTEND="noninteractive" \
          && apt-get update && apt-get upgrade -y --quiet \
          && apt-get install -y --quiet --no-install-recommends \
             nvtop libnvidia-gl-580 libegl1
        RUN  apt-get install -y --quiet --no-install-recommends sudo \
          && echo "ubuntu ALL=(ALL) NOPASSWD: ALL" >/etc/sudoers.d/ubuntu
        RUN  mkdir /home/ubuntu/.cache \
          && chown ubuntu:users /home/ubuntu/.cache
        USER ${UID:-1000}:${GID:-100}

    restart: unless-stopped
    runtime: nvidia
    environment:
      TZ: "Europe/Moscow"
      GGML_VK_VISIBLE_DEVICES: 1,0
      LLAMA_ARG_JINJA: 1
      LLAMA_ARG_HOST: 0.0.0.0
      LLAMA_ARG_PORT: 8033
    command: [
      "--server",
      "--sleep-idle-seconds", "300",
      "--models-max", "1"
    ]
    devices: [ "/dev/dri", "/dev/kfd" ]
    group_add: [ "video", "990" ]  # render == 990
    ipc: host
    privileged: true
    ports:
      - 8033:8033
    volumes:
      - /srv/.models:/home/ubuntu/.cache/llama.cpp:rw

ubuntu@282d79adf616:/app$ ./llama-cli --list-devices
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = NVIDIA GeForce RTX 3090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
load_backend: loaded Vulkan backend from /app/libggml-vulkan.so
load_backend: loaded CPU backend from /app/libggml-cpu-zen4.so
Available devices:
  Vulkan0: Radeon 8060S Graphics (RADV GFX1151) (124928 MiB, 124762 MiB free)
  Vulkan1: NVIDIA GeForce RTX 3090 (24822 MiB, 24329 MiB free)

[686f6c61]

Hey @git2212, @EnderHane, @VVakko, and team,

Had the same problem and these holidays I finally had some time to investigate it properly. Found this discussion and @EnderHane, you nailed it - the missing libglvnd libraries are indeed the culprit.

I did some deeper analysis to understand why: Mesa Vulkan drivers need libglvnd to initialize the EGL display subsystem and communicate with DRI devices (/dev/dri/renderD*). Without these libraries, the ICD loader fails silently and returns zero devices.

I've submitted a PR with the fix: #18664 (tracking in issue #17761)

Tests performed:

• Dependencies check (apt-get check): OK, no conflicts
• Server starts correctly (--help, --version): OK
• Image size: 443MB -> 490MB (+47MB, +10.6%)
• llvmpipe software renderer: OK, no regressions

Before (original Dockerfile):

$ docker run --rm --device /dev/dri:/dev/dri llama-vulkan-original:test --list-devices
ggml_vulkan: No devices found.

After (with fix):

$ docker run --rm vulkan-test-fixed dpkg -l | grep -E 'libglvnd|libegl|libgl'
ii  libegl1:amd64         1.7.0-3    Vendor neutral GL dispatch library -- EGL support
ii  libgl1:amd64          1.7.0-3    Vendor neutral GL dispatch library -- legacy GL support
ii  libglvnd0:amd64       1.7.0-3    Vendor neutral GL dispatch library
ii  libglx0:amd64         1.7.0-3    Vendor neutral GL dispatch library -- GLX support

Thanks for documenting this so well - saved me a lot of time. If anyone has additional tests they'd like me to run, please let me know.

Cheers