Not able to start the docker container

[norey]

Hey All,

My first time trying to use LocalAI and failing to start the container.

…/Projects/localai ❯ docker compose up -d
[+] up 1/2
 ✔ Network localai_default Created                                                                                          0.0s
 ⠋ Container localai-api-1 Starting                                                                                         0.1s
Error response from daemon: could not select device driver "nvidia.com/gpu" with capabilities: [[gpu]]

Compose file:

services:
  api:
    image: localai/localai:latest-gpu-nvidia-cuda-13
    # For CUDA 13, use: localai/localai:latest-gpu-nvidia-cuda-13
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      # start_period, not timeout, is the knob for a slow first boot: startup
      # preload can download tens of GB before the API binds, and failures
      # inside the start period leave the container `starting` rather than
      # marking it unhealthy. timeout is a per-probe deadline.
      start_period: 60m
      interval: 1m
      timeout: 10s
      retries: 3
    ports:
      - 8080:8080
    environment:
      - DEBUG=false
    volumes:
      - ./models:/models:cached
    # CDI driver configuration (recommended for NVIDIA Container Toolkit 1.14+)
    # This uses the nvidia.com/gpu resource API
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia.com/gpu
              count: all
              capabilities: [gpu]

…/Projects/localai ❯ nvidia-ctk -v
NVIDIA Container Toolkit CLI version 1.20.0

…/Projects/localai ❯ nvidia-smi
Tue Aug 25 21:24:08 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.57.04              KMD Version: 610.57.04     CUDA UMD Version: 13.3     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        Off |   00000000:01:00.0 Off |                  N/A |
|  0%   40C    P8              9W /  575W |      66MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

…/Projects/localai ❯ nvidia-ctk cdi list
INFO[0000] Found 3 CDI devices
nvidia.com/gpu=0
nvidia.com/gpu=GPU-2aa233f8-6f8f-d498-0adf-808460be767c
nvidia.com/gpu=all

Hope that enough info.
Thanks.

[ChangkeunJ]

Your host setup looks fine, it's the compose syntax. driver: names a Docker device driver there, not a CDI device, so Docker goes looking for a driver plugin literally called nvidia.com/gpu, finds nothing, and reports that it couldn't select one. The CDI spec you generated never gets consulted at all.

For CDI the device name belongs in device_ids, with the driver set to cdi:

    deploy:
      resources:
        reservations:
          devices:
            - driver: cdi
              device_ids:
                - nvidia.com/gpu=all

Drop count and capabilities while you're in there, they belong to the older shape and don't apply to this one.

If that still doesn't take, the other half is the daemon. CDI landed as an experimental feature in Docker 25 and wants "features": { "cdi": true } in /etc/docker/daemon.json on versions where it isn't on by default, plus a daemon restart. docker info | grep -i cdi will tell you which side of that you're on.

And if you'd rather not deal with CDI at all, the legacy form still works and doesn't touch it:

            - driver: nvidia
              count: all
              capabilities: [gpu]

Your nvidia-ctk cdi list output shows the specs generated correctly, so this really is only about which of the two shapes compose gets handed.

[norey]

I modified the compose file to

deploy:
  resources:
    reservations:
      devices:
        - driver: cdi
          device_ids:
            - nvidia.com/gpu=all

And added "features": { "cdi": true to /etc/docker/daemon.json and restarted the docker service.

❯ docker info | grep -i cdi
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
  cdi: nvidia.com/gpu=0
  cdi: nvidia.com/gpu=GPU-2aa233f8-6f8f-d498-0adf-808460be767c
  cdi: nvidia.com/gpu=all

Now it's another error

# docker compose up -d
validating /home/user/Projects/localai/compose.yml: services.api.deploy.resources.reservations.devices.0 missing property 'capabilities'

capabilities shouldn't have been removed?

# docker -v
Docker version 29.7.2, build a7dcaa6fdb

[ChangkeunJ]

It wasn't removed — it was added. capabilities became a required property of each entry under deploy.resources.reservations.devices in compose-go v2.3.0, which shipped in Compose v2.30.0. Before that the schema had no required key there at all, which is why nearly every example you'll find online omits it. Validation is client-side and runs before anything driver-specific, so naming driver: cdi doesn't exempt you from it.

The value itself doesn't matter in your case. When a device request names a driver, the daemon selects by driver name and skips capability matching entirely, and the CDI handler only ever reads DeviceIDs. So this is enough to get past it:

    deploy:
      resources:
        reservations:
          devices:
            - driver: cdi
              capabilities: ["gpu"]
              device_ids:
                - "nvidia.com/gpu=all"

Don't put count or options next to it — the CDI path rejects both outright ("unexpected count in CDI device request").

There's a shorter route though. Since Compose v2.30.0 a plain service-level devices: ["nvidia.com/gpu=all"] is recognised as a CDI qualified name and turned into a CDI device request directly, so you can drop the deploy: block entirely. Check docker compose version before relying on it — the Compose plugin versions independently of the engine, so 29.7.2 tells you nothing about which one you have. On anything older than 2.30 that line quietly degrades into a host path device mapping and you get no such file or directory at container start instead of a validation error.

The "features": {"cdi": true} in daemon.json can go too if you want. CDI has been enabled by default since Engine 28.2.0, so on 29.7.2 it's a no-op — harmless to leave in either way.

[norey]

This works fine, thanks!
I guess https://localai.io/docs/installation/containers/#using-cdi-container-device-interface---recommended-for-nvidia-container-toolkit-114 needs updating.