llama.cpp Docker Server

A production-ready Docker setup for running llama.cpp models with GPU acceleration using CUDA. Supports multiple models running concurrently on different ports with automatic driver detection and CUDA version management.

Prerequisites

• Docker Engine (20.10+)
• NVIDIA GPU with compute capability 6.0 or higher
• NVIDIA Docker Runtime (nvidia-docker package)
• NVIDIA Driver (recent version recommended)

Verify NVIDIA Docker runtime:

docker run --rm --gpus all nvidia/cuda:12.4.0-runtime-ubuntu22.04 nvidia-smi

Quick Start

1. Make scripts executable

chmod +x build_server_cuda.sh run_model.sh

2. Build the Docker image

cp .env.example .env
./build_server_cuda.sh

The build script automatically detects your NVIDIA driver and selects the appropriate CUDA version (12.4 or 13.0). This creates the image local/llama.cpp:server-cuda which is reused for all model instances.

3. Run a model

./run_model.sh unsloth/Qwen3-VL-8B-Instruct-GGUF:Q5_K_M 7000

The model is now available at http://localhost:7000.

Usage

Running Models

Start a model with:

./run_model.sh <MODEL_NAME> <PORT> [CONTEXT_SIZE]

Parameters:

• MODEL_NAME: Hugging Face model identifier or GGUF repository
• PORT: Port to expose the server (e.g., 7000, 7001)
• CONTEXT_SIZE: Optional, context window size (default: 32768)

Examples:

# 8B Qwen3 Vision model on port 7000
./run_model.sh unsloth/Qwen3-VL-8B-Instruct-GGUF:Q5_K_M 7000

# 8B LLaMA-3.1 with 16KB context on port 7001
./run_model.sh meta-llama/Llama-3.1-8B-Instruct 7001 16384

# 70B model with extended context on port 7002
./run_model.sh mistralai/Mistral-7B-Instruct 7002 32768

Each model runs in a dedicated container and shares the same Docker image. Multiple models can run simultaneously on different ports without rebuilding.

API Requests

Once a model is running, send requests to the OpenAI-compatible API endpoint:

curl http://localhost:7000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3",
    "messages": [
      {
        "role": "user",
        "content": "Explain quantum computing in one paragraph."
      }
    ],
    "temperature": 0.7,
    "max_tokens": 256
  }'

Response:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "qwen3",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Quantum computing..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 45,
    "total_tokens": 57
  }
}

Configuration

Environment Variables

Edit .env to customize build behavior:

cp .env.example .env

• UBUNTU_VERSION: Base Ubuntu image version (default: 22.04)
• CUDA_VERSION: CUDA toolkit version (auto-detected by build script)

The build script automatically selects CUDA 12.4 or 13.0 based on your NVIDIA driver version. Manual override is not required in most cases.

Model Storage

Downloaded models are cached in ./.models/ directory to avoid re-downloading:

ls ./.models/
# Output: models downloaded and cached by llama.cpp

Deployment

Docker Compose (Production)

For production deployments, use Docker Compose to orchestrate multiple models:

# Build image
docker compose --profile build up

# Run multiple models
docker compose --profile runtime up -d llama-runtime

Override runtime parameters with environment variables:

MODEL=unsloth/Qwen3-VL-8B-Instruct-GGUF:Q5_K_M \
PORT=7000 \
CTX_SIZE=32768 \
./run_model.sh

Monitoring

Check running containers:

docker ps --filter "ancestor=local/llama.cpp:server-cuda"

Check container logs:

docker logs <container-id>

View GPU usage:

docker exec <container-id> nvidia-smi

Troubleshooting

GPU not detected:

docker run --rm --gpus all nvidia/cuda:12.4.0-runtime-ubuntu22.04 nvidia-smi

CUDA version mismatch: The build script auto-detects and selects the correct CUDA version. If manual override is needed, edit .env before building.

Model download timeout: Large models may take time to download. Check container logs:

docker logs <container-id> | tail -20

Port already in use: Specify a different port when running the model:

./run_model.sh unsloth/Qwen3-VL-8B-Instruct-GGUF:Q5_K_M 7001

Architecture

• Build Stage: Compiles llama.cpp with CUDA support using GPU-optimized flags
• Runtime Stage: Lean CUDA runtime container that loads pre-built binaries
• Multi-Model: Each model instance runs in its own container, isolated and scalable
• Shared Cache: Model files are downloaded once and reused across containers

Performance Notes

• First model download may take 5-30 minutes depending on model size and network
• Subsequent runs of the same model use cached files
• GPU memory requirements vary by model size (8B ≈ 6-8GB, 70B ≈ 40-48GB)
• Adjust context size to fit available GPU memory