docker‐compose.yml for AMD GPUs

A docker-compose.yml for AMD-GPUs with reload on config change.

In this example, a config.yaml (.yaml, not .yml) file is expected in a subfolder “config”. We pass the subfolder, else auto-reload on config change won’t work.

Example structure:
/home/docker/llama-swap/docker-compose.yml
/home/docker/llama-swap/config/config.yaml
/home/docker/llama-swap/models

services:
  llama-swap:
    image: ghcr.io/mostlygeek/llama-swap:vulkan
    pull_policy: always
    container_name: llama-swap
    devices:
      - /dev/dri:/dev/dri
      - /dev/kfd:/dev/kfd
    group_add:
      - "video"
#    security_opt:     # onyl needed if “operation not permitted” when the container tries to access GPU resources
#      - seccomp=unconfined
    shm_size: 2g       # optional
    volumes:
      - ./models:/root/.cache/llama.cpp/ # location where your models are stored, change left of ":" to change 
      - ./config:/app/config # location of your config file. we pass a folder for auto-reloading on config change. Change left of ":" to change, but pass a folder, not the file
    ports:
      - 8080:8080      # Change left of ":" to change the port llama-swap will be accessible on
    restart: unless-stopped
    entrypoint: ["/app/llama-swap", "-config", "/app/config/config.yaml", "--watch-config"]