Ollama doesn't detect the GPU (requires `--gpus==all` instead of `--runtime=nvidia`)

[peterschmidt85]

If we run the ollama/ollama Docker image via dstack (any type of the configuration), it doesn't detect GPU.

type: service

image: ollama/ollama
commands:
  - ollama serve &
  - sleep 3
  - ollama pull mixtral
port: 11434

resources:
  gpu: 48GB..80GB

model:
  type: chat
  name: mixtral
  format: openai

time=2024-02-16T15:55:33.866Z level=INFO source=images.go:706 msg="total blobs: 0"
time=2024-02-16T15:55:33.866Z level=INFO source=images.go:713 msg="total unused blobs removed: 0"
time=2024-02-16T15:55:33.866Z level=INFO source=routes.go:1014 msg="Listening on [::]:11434 (version 0.1.25)"
time=2024-02-16T15:55:33.867Z level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
root@6abfea53-98bd-4c93-8cbe-3887fe8a3b42:/workflow# time=2024-02-16T15:55:38.877Z level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [cuda_v11 rocm_v5 cpu cpu_avx cpu_avx2 rocm_v6]"
time=2024-02-16T15:55:38.877Z level=INFO source=gpu.go:94 msg="Detecting GPU type"
time=2024-02-16T15:55:38.877Z level=INFO source=gpu.go:262 msg="Searching for GPU management library libnvidia-ml.so"
time=2024-02-16T15:55:38.880Z level=INFO source=gpu.go:308 msg="Discovered GPU libraries: []"
time=2024-02-16T15:55:38.880Z level=INFO source=gpu.go:262 msg="Searching for GPU management library librocm_smi64.so"
time=2024-02-16T15:55:38.880Z level=INFO source=gpu.go:308 msg="Discovered GPU libraries: []"
time=2024-02-16T15:55:38.880Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-16T15:55:38.881Z level=INFO source=routes.go:1037 msg="no GPU detected"

If we try to do the same but instead of --runtime=nvidia, use --gpus==all, everything works well.

time=2024-02-16T15:58:37.126Z level=INFO source=images.go:706 msg="total blobs: 0"
time=2024-02-16T15:58:37.126Z level=INFO source=images.go:713 msg="total unused blobs removed: 0"
time=2024-02-16T15:58:37.127Z level=INFO source=routes.go:1014 msg="Listening on [::]:11434 (version 0.1.25)"
time=2024-02-16T15:58:37.128Z level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
time=2024-02-16T15:58:42.648Z level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [cpu cpu_avx2 rocm_v5 rocm_v6 cuda_v11 cpu_avx]"
time=2024-02-16T15:58:42.649Z level=INFO source=gpu.go:94 msg="Detecting GPU type"
time=2024-02-16T15:58:42.649Z level=INFO source=gpu.go:262 msg="Searching for GPU management library libnvidia-ml.so"
time=2024-02-16T15:58:42.649Z level=INFO source=gpu.go:308 msg="Discovered GPU libraries: [/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.535.113.01]"
time=2024-02-16T15:58:42.660Z level=INFO source=gpu.go:99 msg="Nvidia GPU detected"
time=2024-02-16T15:58:42.660Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-16T15:58:42.667Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9"

Additional information: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4585