Misc. bug: llama.cpp:server-cuda13 Docker image fails to run on supported GPU

[FlorinAndrei]

Name and Version

$ docker run -it --rm ghcr.io/ggml-org/llama.cpp:server-cuda13 --version
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
version: 8982 (b42c7fa)
built with GNU 14.2.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

docker run --rm --gpus all \
  -v ~/models:/models \
  -p 8080:8080 \
  ghcr.io/ggml-org/llama.cpp:server-cuda13 \
  --models-preset /models/config.ini \
  --port 8080 --host 0.0.0.0 \
  --threads -1 \
  --threads-batch -1 \
  --no-warmup \
  --sleep-idle-seconds 300 \
  --models-max 1 \
  --parallel 1 \
  --kv-unified \
  --cache-idle-slots

Problem description & steps to reproduce

This is config.ini:

version = 1

[*]
n-gpu-layers     = all
flash-attn       = on
cache-type-k     = q8_0
cache-type-v     = q8_0

[Qwen3.6-27B-Q4_K_M-general]
model            = /models/unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q4_K_M.gguf
temp             = 1.0
top-k            = 20
top-p            = 0.95
min-p            = 0.0
presence-penalty = 0.0
repeat-penalty   = 1.0
chat-template-kwargs = {"enable_thinking": true}

[Qwen3.6-27B-Q4_K_M-coding]
model            = /models/unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q4_K_M.gguf
temp             = 0.6
top-k            = 20
top-p            = 0.95
min-p            = 0.0
presence-penalty = 0.0
repeat-penalty   = 1.0
chat-template-kwargs = {"enable_thinking": true}

[Qwen3.6-27B-Q4_K_M-nothink]
model            = /models/unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q4_K_M.gguf
temp             = 0.7
top-k            = 20
top-p            = 0.8
min-p            = 0.0
presence-penalty = 0.0
repeat-penalty   = 1.0
chat-template-kwargs = {"enable_thinking": false}

[gemma-4-26B-A4B-it-Q4_K_M]
model            = /models/ggml-org/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-Q4_K_M.gguf
temp             = 1.0
top-k            = 64
top-p            = 0.95

The machine is a home PC with an AMD Ryzen 7 CPU and an RTX 3090 GPU running Ubuntu 24.04. Here is the relevant GPU-related info:

Driver: 580.142
CUDA (driver max): 13.0
GPU: NVIDIA GeForce RTX 3090
Compute cap: 8.6

I run apps using CUDA 13 on this machine all the time. No problem. In fact, I don't think I normally run older CUDA versions anymore.

But the ghcr.io/ggml-org/llama.cpp:server-cuda13 fails to use the GPU. All models run on the CPU instead. This is weird because the compute capability should definitely be supported.

The ghcr.io/ggml-org/llama.cpp:server-cuda works okay on the same machine, and is able to use the GPU.

First Bad Commit

No response

Relevant log output

Logs

ggml_cuda_init: failed to initialize CUDA: forward compatibility was attempted on non supported HW
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
build_info: b8982-b42c7fa5b
system_info: n_threads = 16 (n_threads_batch = 16) / 16 | CUDA : ARCHS = 750,800,860,890,1200,1210 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
Running without SSL
init: using 15 threads for HTTP server
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
srv   load_models: Loaded 0 cached model presets
srv   load_models: Loaded 5 custom model presets from /models/config.ini
srv   load_models: Available models (5) (*: custom preset)
srv   load_models:   * Qwen3.6-27B-Q4_K_M-coding
srv   load_models:   * Qwen3.6-27B-Q4_K_M-general
srv   load_models:   * Qwen3.6-27B-Q4_K_M-nothink
srv   load_models:   * default
srv   load_models:   * gemma-4-26B-A4B-it-Q4_K_M
main: starting router server, no model will be loaded in this process
start: binding port with default address family
main: router server is listening on http://0.0.0.0:8080
main: NOTE: router mode is experimental
main:       it is not recommended to use this mode in untrusted environments

[CISC]

Driver: 580.142
CUDA (driver max): 13.0
GPU: NVIDIA GeForce RTX 3090
Compute cap: 8.6

You probably need to update your driver to a version that supports CUDA 13.1.

[FlorinAndrei]

You probably need to update your driver to a version that supports CUDA 13.1.

Two observations:

The driver I have is the default in Ubuntu 24.04. It's not impossible to install another version, but I just want to point that out.

On a DGX Spark system (ARM CPU, Blackwell GPU, unified memory) ghcr.io/ggml-org/llama.cpp:server-cuda13 works just fine, despite that machine having the exact same driver version (and for the same reason, it's the default). Here's the Spark machine:

$ nvidia-smi
Thu Apr 30 16:19:26 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.142                Driver Version: 580.142        CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    Off |   0000000F:01:00.0  On |                  N/A |
| N/A   42C    P0             10W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2660      G   /usr/lib/xorg/Xorg                       43MiB |
|    0   N/A  N/A            3481      C   /usr/local/bin/python3                  274MiB |
|    0   N/A  N/A            5221      G   /usr/bin/gnome-shell                     16MiB |
+-----------------------------------------------------------------------------------------+

[CISC]

The driver I have is the default in Ubuntu 24.04. It's not impossible to install another version, but I just want to point that out.

Not the default, but possibly the current stable?

Either way, it's not uncommon (but somewhat painful, I get that) to install from https://developer.nvidia.com/cuda-downloads

On a DGX Spark system (ARM CPU, Blackwell GPU, unified memory) ghcr.io/ggml-org/llama.cpp:server-cuda13 works just fine, despite that machine having the exact same driver version (and for the same reason, it's the default). Here's the Spark machine:

If so there's no reason for it not to work for you too I guess, try to figure out why it doesn't. :)

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.