LLM models fail to download (everything else downloads fine)

[stepanzel]

Hello guys, I feel I'm coming closer to getting this project off the ground. Now I'm facing this error here (llama3.1:8b-text-q2_K 3.2GB):
[ info ] [model-downloads] Processing job: 16cdc0f10ed915d5 of type: download-model [ error ] [model-downloads] Job failed: 16cdc0f10ed915d5, Error: Failed to initiate download for model: Failed to download model: EOF

and for a different model (deepseek-r1:1.5b 20GB):
[ info ] [model-downloads] Processing job: 5b6e7b956375b6a7 of type: download-model [ error ] [model-downloads] Job failed: 5b6e7b956375b6a7, Error: Failed to initiate download for model: Failed to download model: write /root/.ollama/models/blobs/sha256-6150cb382311b69f09cc0f9a1b69fc029cbd742b66bb8ec531aa5ecf5c613e93-partial-0: no space left on device
even though I'm using only 600GB out of 1.55TB.

How can I dig deeper what the error might be (i. e. enable verbose logging)? I tried force-reinstalling the AI assistant to no avail.

Thanks!

Edit: after investigating the container and its storage parameters, I might have found a clue: the host path in ollama is "/opt/project-nomad/storage/ollama" while nomad_admin for example is using "/NOMAD/storage". How can I adjust the storage parameters for the containers downloaded within NOMAD itself? Or could I add ollama and kiwix to the Docker configuration file directly?

[louie-lab2000]

Models won't download if you don't have a Ollama compatible GPU installed. What type of GPU are you using and how are you making it available to the application stack? In my case, I am using an Nvidia card, and I had to install the Nvidia driver and toolkit to my docker VM after passing through my GPU to the same docker VM.

[louie-lab2000]

Here's the compose file I used. Notice I didn't add Ollama or Kiwix to this, but I did add a deploy block to the admin service to make my Nvidia GPU available to the stack

name: project-nomad

services:
  admin:
    image: ghcr.io/crosstalk-solutions/project-nomad:latest
    pull_policy: always
    container_name: nomad_admin
    restart: unless-stopped
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "8080:8080"
    volumes:
      - /opt/project-nomad/storage:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - nomad-update-shared:/app/update-shared
    environment:
      - NODE_ENV=production
      - PORT=8080
      - LOG_LEVEL=info
      - APP_KEY=changeme
      - HOST=0.0.0.0
      - URL=http://yoururl:8080
      - DB_HOST=mysql
      - DB_PORT=3306
      - DB_DATABASE=nomad
      - DB_USER=nomad_user
      - DB_PASSWORD=changeme
      - DB_NAME=nomad
      - DB_SSL=false
      - REDIS_HOST=redis
      - REDIS_PORT=6379
    depends_on:
      mysql:
        condition: service_healthy
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/api/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    # GPU passthrough — allows Nomad to detect the GPU and configure Ollama with it.
    # Requires nvidia-container-toolkit installed on the host.
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  dozzle:
    image: amir20/dozzle:v10.0
    container_name: nomad_dozzle
    restart: unless-stopped
    ports:
      - "9999:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DOZZLE_ENABLE_ACTIONS=false
      - DOZZLE_ENABLE_SHELL=false

  mysql:
    image: mysql:8.0
    container_name: nomad_mysql
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=changeme
      - MYSQL_DATABASE=nomad
      - MYSQL_USER=nomad_user
      - MYSQL_PASSWORD=changeme
    volumes:
      - /opt/project-nomad/mysql:/var/lib/mysql
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      interval: 30s
      timeout: 10s
      retries: 10

  redis:
    image: redis:7-alpine
    container_name: nomad_redis
    restart: unless-stopped
    volumes:
      - /opt/project-nomad/redis:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 30s
      timeout: 10s
      retries: 3

  updater:
    image: ghcr.io/crosstalk-solutions/project-nomad-sidecar-updater:latest
    pull_policy: always
    container_name: nomad_updater
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /opt/project-nomad:/opt/project-nomad
      - nomad-update-shared:/shared

  disk-collector:
    image: ghcr.io/crosstalk-solutions/project-nomad-disk-collector:latest
    pull_policy: always
    container_name: nomad_disk_collector
    restart: unless-stopped
    volumes:
      - /:/host:ro,rslave
      - /opt/project-nomad/storage:/storage

volumes:
  nomad-update-shared:
    driver: local

[louie-lab2000]

Sorry for three responses to this, but if you need to know how to install the Nvidia drivers and toolkit, here's what I did

Nvidia GPU Setup on Debian for Docker

1. Enable non-free repos and install drivers:

sudo sed -i 's/main/main contrib non-free non-free-firmware/' /etc/apt/sources.list
sudo apt update
sudo apt install -y linux-headers-amd64 nvidia-driver

2. Add NVIDIA Container Toolkit repo:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

3. Install toolkit and configure Docker:

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

4. Verify:

sudo nvidia-smi
sudo docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

[stepanzel]

Thank you for your response!
I found out that this problem impacts not only llama but also kiwix and qdrant; they all use the "wrong" storage path. So I doubt it's because the Nvidia driver.
To answer your questions though, I don't have a graphics card yet, want to try and see the performance using the CPU only. And if that's unbearable, at least I should have the kiwix libraries available.

I will try the workaround mentioned here: #390 (comment) and will report back.

[louie-lab2000]

Ollama in this stack, the way its implemented, will not download any models if you don't have a GPU. I tried it.

[stepanzel]

Sorry, have to contradict you:
a) I used the symlink workaround and suddenly everything can be downloaded, yay!
b) I'm hosting the app on a QNAP NAS without GPU and ollama still runs, slowly though but it can run! Now I just need to find a model which can deliver a compromise between speed and accuracy.