Docker Compose for Large MoE Models

Docker Compose for large MoE models

Build moe-cache in a CUDA container and mount GGUF shards read-only from the host. The image never contains the model.

Use moe-cache commit 925933801 or newer. The example uses Qwen3.8 Flash Next UD-Q3_K_XL, cache64, 65,536-token Q8 KV, and lazy mode.

Important

WINDOWS USERS: Docker Desktop has different networking and memory controls. Use Windows WDDM setup for a native Windows executable. The owner validated this container recipe only on native Linux.

Install the host requirements

Install these components:

• A current NVIDIA driver
• Docker Engine
• Docker Compose
• Docker Buildx
• NVIDIA Container Toolkit

On Arch Linux or CachyOS:

sudo pacman -Syu docker docker-compose docker-buildx nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl enable --now docker
sudo systemctl restart docker
sudo usermod -aG docker "$USER"
newgrp docker

newgrp docker starts a child shell with the new group immediately. A complete logout and login does the same for future sessions. Membership in the docker group grants root-equivalent access to the Docker daemon.

For Ubuntu and other distributions, use the Docker Engine installation guide and the NVIDIA Container Toolkit installation guide. Make sure the Docker packages include the Compose and Buildx plugins.

Verify GPU access before building llama.cpp:

docker run --rm --gpus all nvidia/cuda:12.8.1-base-ubuntu24.04 nvidia-smi

Do not continue until this command shows the host GPU.

Clone the fork and select the branch

git clone --branch moe-cache --single-branch https://github.com/GenerelSchwerz/llama.cpp.git
cd llama.cpp

Create the environment file

Create .env in the repository root:

MODEL_DIR=/absolute/path/to/Qwen3.8-Flash-Next-GGUF/UD-Q3_K_XL
MODEL_FILE=Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf
CUDA_DOCKER_ARCH=120

Use an absolute MODEL_DIR. Keep every shard in that directory and point MODEL_FILE at the first shard. llama.cpp discovers the other shards.

Common CUDA architecture values are 120 for RTX 50 series, 89 for RTX 40 series, and 86 for RTX 30 series. Use default when building one image for multiple GPU generations, at the cost of a slower build and larger build workload.

Create compose.yaml

name: llama-moe-cache

services:
  llama-server:
    image: local/llama.cpp:moe-cache-server-cuda
    build:
      context: .
      dockerfile: .devops/cuda.Dockerfile
      target: server
      network: host
      args:
        CUDA_VERSION: "12.8.1"
        CUDA_DOCKER_ARCH: ${CUDA_DOCKER_ARCH:-default}
    gpus: all
    network_mode: host
    volumes:
      - ${MODEL_DIR}:/models:ro
    command:
      - --model
      - /models/${MODEL_FILE}
      - --host
      - 127.0.0.1
      - --port
      - "8080"
      - --n-gpu-layers
      - all
      - --ctx-size
      - "65536"
      - --parallel
      - "1"
      - --batch-size
      - "8192"
      - --ubatch-size
      - "512"
      - --threads
      - "12"
      - --cache-type-k
      - q8_0
      - --cache-type-v
      - q8_0
      - --kv-offload
      - --moe-expert-cache-size
      - "64"
      - --load-mode
      - none
      - --lazy-mode
      - "on"
      - --flash-attn
      - "on"
      - --fit
      - "off"
      - --cache-ram
      - "0"
      - --ctx-checkpoints
      - "0"
      - --jinja
      - --no-warmup
      - --metrics
    healthcheck:
      test: ["CMD", "curl", "-f", "http://127.0.0.1:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 30
      start_period: 180s
    stop_grace_period: 30s

The branch supports the Dockerfile's standard dynamically loaded CUDA backend. Do not add a custom static-backend build or put the model in the image.

build.network: host avoids build-time DNS failures on Linux hosts where Cloudflare WARP or another local resolver exposes only loopback DNS addresses. It affects image construction only. Remove it if local policy prohibits host networking and Docker bridge DNS already works.

The service uses host networking because WARP broke bridge DNS during validation. On Docker Desktop, remove network_mode: host and publish the port unless Docker Desktop host networking is enabled:

    ports:
      - 127.0.0.1:8080:8080

Also change the server --host value to 0.0.0.0 when using port publication.

Build and start

docker compose config --quiet
docker compose build llama-server
docker compose up -d --no-build
docker compose logs -f llama-server

The first build can take several minutes. A small build context confirms that the external model is not being sent to Docker. Model loading can still take a minute or more because CUDA pools, the KV cache, and host-backed expert sources must be initialized.

Wait until the log says model loaded, then verify both health and generation:

curl --fail http://127.0.0.1:8080/health

curl --fail-with-body http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Reply with one short sentence."}],"max_tokens":128,"temperature":0}'

Health proves that the HTTP process is ready. The chat request proves that prompt evaluation and token generation work. Check memory while the request runs:

docker stats --no-stream
nvidia-smi

Stop and remove the container without deleting the image or host model:

docker compose down

Large-model memory rules

• Do not set mem_limit unless it is above the measured container working set. A container limit can cause OOMKilled=true even when the host has free RAM or swap.
• --lazy-mode on does not mean zero host-memory use. This configuration used about 53.5 GiB after a short request.
• The 64-slot cache plus 64K Q8 KV configuration used about 13.9 GiB process VRAM on the validated RTX 5070 Ti. Leave room for the display server and driver allocations.
• Put lazy-read model shards on SSD or NVMe. HDD latency can dominate cache misses.
• Do not put the GGUF files inside the repository. The bind mount keeps the 90 GB model outside the approximately 2.5 GB image and avoids copying it during every rebuild.
• A larger context reserves more KV memory even when the current prompt is short. Reduce context before assuming Docker itself is the problem.

Troubleshooting

One very large CUDA allocation during load

If the log asks for tens of GiB in one CUDA allocation, update the branch and rebuild without cached source layers:

git pull --ff-only origin moe-cache
docker compose build --no-cache llama-server

For this Qwen model, a 57,697 MiB allocation identified an old image without dynamic-backend MoE cache discovery. Rebuild from moe-cache and confirm that the command includes --moe-expert-cache-size.

Container exits or is killed during load

docker compose ps -a
docker inspect llama-moe-cache-llama-server-1 \
  --format 'OOMKilled={{.State.OOMKilled}} Exit={{.State.ExitCode}}'
docker compose logs --tail 300 llama-server

If OOMKilled=true, remove container memory limits or increase the RAM assigned to Docker Desktop or WSL. On native Linux, check host RAM and swap. The host must support the cold expert source, other host tensors, runtime buffers, and the operating system.

CUDA out of memory during startup or generation

Reduce --moe-expert-cache-size to 48 or 32. If necessary, reduce --ctx-size, --batch-size, or --ubatch-size. This is GPU allocation pressure, not evidence that the GGUF must be copied into the image.

Model file or shard errors

Check the resolved mount and file name:

docker compose config
docker compose run --rm --entrypoint ls llama-server -lh /models

All shards must be visible inside /models. File names are case-sensitive.

GPU is unavailable in Compose

Repeat the CUDA nvidia-smi smoke test. If it fails, configure and restart the runtime:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Build hangs at npm ci with EAI_AGAIN

Keep build.network: host on native Linux when Cloudflare WARP breaks Docker build DNS, then rebuild.

Published localhost port resets while the internal health check passes

On native Linux, use network_mode: host as shown above. This bypasses the Docker bridge and its NAT path. Bind llama-server to 127.0.0.1 unless remote access is intentional.

Validated result

The owner validation used the standard dynamic-backend CUDA Dockerfile built from moe-cache commit 12a4d1d01. The final reviewed image reached health in 73 seconds. A 61-token prompt ran at 67.73 tokens/s and a 38-token decode ran at 36.16 tokens/s. Grouped MoE telemetry reported 1,776 completed operations, zero fallbacks, and zero errors. The loaded container used 53.14 GiB RAM; after the request it used 53.50 GiB RAM and 13,872 MiB process VRAM. The exact requested answer was returned, the container reported OOMKilled=false, and teardown exited cleanly and closed port 8080.

MoE cache overview - MoE cache flags - 16 GB setup - Hardware guides - Home