Skip to content

Ollama Setup Guide

Jump to bottom
Andy Gillis edited this page Jul 1, 2026 · 1 revision

Ollama Setup Guide (WSL2 + Docker + GPU)

Introduction

Ollama is a local large language model runtime that allows running models such as Llama 3.1, Qwen, DeepSeek, Phi, and others directly on your GPU.
This guide covers installation, GPU passthrough, DNS stability, Docker configuration, model management, and testing.

Prerequisites

  • Windows 11
  • WSL2 with Ubuntu
  • NVIDIA GPU
  • Latest Windows NVIDIA driver
  • Docker installed inside WSL2
  • NVIDIA Container Toolkit installed
  • Stable DNS configuration
  • Basic familiarity with Docker Compose

Step 1 – Verify NVIDIA GPU inside WSL2

Check that WSL2 can see your GPU.

nvidia-smi

If your GPU appears, continue.
If not, update your Windows NVIDIA driver.

Step 2 – Install Docker inside WSL2

Install Docker directly inside Ubuntu.

sudo apt update
sudo apt install docker.io docker-compose-plugin -y

Enable Docker service.

sudo systemctl enable --now docker

Add your user to the Docker group.

sudo usermod -aG docker $USER

Restart WSL.

Step 3 – Install NVIDIA Container Toolkit

This enables GPU passthrough into Docker containers.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/ubuntu22.04/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit

Configure Docker to use the NVIDIA runtime.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test GPU passthrough.

docker run --rm --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi

If your GPU appears, passthrough is working.

Step 4 – Fix DNS for Docker

WSL2 and Docker sometimes use slow or broken DNS.
Add DNS servers directly to your Docker Compose file.

dns:
  - 1.1.1.1
  - 8.8.8.8

This ensures fast model downloads and stable registry access.

Step 5 – Example Ollama Docker Compose Service

Below is a working example of an Ollama service with GPU passthrough and DNS overrides.

services:
  ollama:
    container_name: ollama.${HOST_NAME}
    hostname: ollama.${HOST_NAME}.lan
    image: ollama/ollama:latest
    gpus: all
    environment:
      TZ: ${TZ}
      OLLAMA_FLASH_ATTENTION: "true"
      OLLAMA_NUM_PARALLEL: "1"
      OLLAMA_KV_CACHE_TYPE: "q8_0"
      OLLAMA_MAX_LOADED_MODELS: "1"
      OLLAMA_CONTEXT_LENGTH: "8192"
    networks:
      - mediaserver
    ports:
      - "${OLLAMA_PORT}:11434"
    dns:
      - 1.1.1.1
      - 8.8.8.8
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ${DOCKERDIR}/ollama:/root/.ollama
    restart: always
    security_opt:
      - no-new-privileges:true
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
      - "homepage.group=Infrastructure"
      - "homepage.name=Ollama"
      - "homepage.icon=ollama.png"
      - "homepage.href=https://ollama.${DOMAINNAME}/"
      - "homepage.description=Local LLM engine"

Start the stack.

docker compose up -d

Step 6 – Verify GPU Detection in Ollama

Check Ollama logs for CUDA.

docker logs ollama.<HOST_NAME> | grep -i CUDA

You should see a line indicating CUDA and your GPU model.

Step 7 – Pulling Models

Pull a model.

docker exec -it ollama.<HOST_NAME> ollama pull llama3.1:8b

List installed models.

docker exec -it ollama.<HOST_NAME> ollama list

Remove a model.

docker exec -it ollama.<HOST_NAME> ollama rm llama3:8b

Step 8 – Testing GPU Inference

Run a test prompt.

docker exec -it ollama.<HOST_NAME> ollama run llama3.1:8b "Say hello and confirm GPU usage."

Monitor GPU usage.

watch -n 1 nvidia-smi

You should see

  • VRAM increase
  • GPU utilization increase
  • Power usage increase

This confirms GPU inference.

Step 9 – Connecting OpenWebUI to Ollama

OpenWebUI connects to Ollama using the API endpoint.

Example endpoint
http://ollama.<HOST_NAME>:11434

Test connectivity from inside the OpenWebUI container.

docker exec -it openwebui.<HOST_NAME> curl http://ollama.<HOST_NAME>:11434/api/tags

If you see your model list, the connection is working.

Backup and Restore

Backup

Backup the Ollama data directory.

./appdata/ollama

This contains downloaded models and metadata.

Restore

Copy the directory into a new Ollama container volume and restart the stack.

Troubleshooting

Slow model downloads

Ensure DNS is set in Docker Compose.

dns:
  - 1.1.1.1
  - 8.8.8.8

GPU not detected

Test passthrough.

docker run --rm --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi

Ollama using CPU instead of GPU

Check for Vulkan fallback.

docker logs ollama.<HOST_NAME> | grep -i vulkan

If Vulkan appears, CUDA is not active.

WSL DNS corruption

Reset DNS.

sudo rm /etc/resolv.conf
echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf
sudo chattr +i /etc/resolv.conf

Notes

  • Llama 3.1 8B is the best model for GPUs with 12 GB VRAM
  • Larger models such as 70B run CPU‑only
  • DNS stability is critical for fast model downloads
  • GPU inference requires NVIDIA Container Toolkit and correct Docker configuration

Clone this wiki locally