Older CUDA compute capability 3.5 and 3.7 support

[orlyandico]

I recently put together an (old) physical machine with an Nvidia K80, which is only supported up to CUDA 11.4 and Nvidia driver 470. All my previous experiments with Ollama were with more modern GPU's.

I found that Ollama doesn't use the GPU at all. I cannot find any documentation on the minimum required CUDA version, and if it is possible to run on older CUDA versions (e.g. Nvidia K80, V100 are still present on cloud, e.g. G2 and P2 on AWS) and there's lots of K80's all over ebay.

EDIT: looking through the logs, it appears that the GPU's are being seen:

Jan 1 20:22:43 thinkstation-s30 ollama[911]: 2024/01/01 20:22:43 llama.go:300: 24762 MB VRAM available, loading up to 162 GPU layers
Jan 1 20:22:43 thinkstation-s30 ollama[911]: 2024/01/01 20:22:43 llama.go:436: starting llama runner
Jan 1 20:22:43 thinkstation-s30 ollama[911]: 2024/01/01 20:22:43 llama.go:494: waiting for llama runner to start responding
Jan 1 20:22:43 thinkstation-s30 ollama[911]: ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
Jan 1 20:22:43 thinkstation-s30 ollama[911]: ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
Jan 1 20:22:43 thinkstation-s30 ollama[911]: ggml_init_cublas: found 3 CUDA devices:
Jan 1 20:22:43 thinkstation-s30 ollama[911]: Device 0: Tesla K80, compute capability 3.7
Jan 1 20:22:43 thinkstation-s30 ollama[911]: Device 1: Tesla K80, compute capability 3.7
Jan 1 20:22:43 thinkstation-s30 ollama[911]: Device 2: NVIDIA GeForce GT 730, compute capability 3.5

and

Jan 1 20:34:20 thinkstation-s30 ollama[911]: llm_load_tensors: ggml ctx size = 0.11 MiB
Jan 1 20:34:20 thinkstation-s30 ollama[911]: llm_load_tensors: using CUDA for GPU acceleration
Jan 1 20:34:20 thinkstation-s30 ollama[911]: llm_load_tensors: mem required = 70.46 MiB
Jan 1 20:34:20 thinkstation-s30 ollama[911]: llm_load_tensors: offloading 32 repeating layers to GPU
Jan 1 20:34:20 thinkstation-s30 ollama[911]: llm_load_tensors: offloading non-repeating layers to GPU
Jan 1 20:34:20 thinkstation-s30 ollama[911]: llm_load_tensors: offloaded 33/33 layers to GPU
Jan 1 20:34:20 thinkstation-s30 ollama[911]: llm_load_tensors: VRAM used: 3577.61 MiB

but....

Jan 1 20:34:21 thinkstation-s30 ollama[911]: CUDA error 209 at /go/src/github.com/jmorganca/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:7801: no kernel image is available for execution on the device
Jan 1 20:34:21 thinkstation-s30 ollama[911]: current device: 0
Jan 1 20:34:21 thinkstation-s30 ollama[911]: GGML_ASSERT: /go/src/github.com/jmorganca/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:7801: !"CUDA error"
Jan 1 20:34:22 thinkstation-s30 ollama[911]: 2024/01/01 20:34:22 llama.go:451: 209 at /go/src/github.com/jmorganca/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:7801: no kernel image is available for execution on the device
Jan 1 20:34:22 thinkstation-s30 ollama[911]: current device: 0
Jan 1 20:34:22 thinkstation-s30 ollama[911]: GGML_ASSERT: /go/src/github.com/jmorganca/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:7801: !"CUDA error"
Jan 1 20:34:22 thinkstation-s30 ollama[911]: 2024/01/01 20:34:22 llama.go:459: error starting llama runner: llama runner process has terminated

[Cybervet]

What is your Linux Kernel? I think 6+ kernels don't support a lot of older nvidia cards.

[orlyandico]

Kernel is 6+ and the setup is supported. I was able to get PyTorch working with CUDA - albeit PyTorch 2.0.1 only since that is the last version that supports CUDA 11.4

The error 209 "no kernel image is available for execution on the device" is for CUDA, not the Linux kernel. Basically the Ollama distribution doesn't have a compiled kernel (via nvcc) for CUDA 11.4 (not even sure if that is supported, if I build from source).

[yolobnb]

This is also the same case for me. I am using a Quadro K2200. It is recognized along with the computing capability. As soon as I pull a model, the error shows up and Ollama terminates.

[dhiltgen]

The K80 is Compute Capability 3.7, which at present isn't supported by our CUDA builds. (see https://developer.nvidia.com/cuda-gpus for the mapping table)

Based on our current build setup, Compute Capability 6.0 is the minimum we'll support. We had some bugs on detection and fallback logic in 0.1.18, which should be resolved in 0.1.19 so that if we detect older than 6.0 we'll fallback to CPU.

There's a possibility we may be able to support 5.x cards by compiling llama.cpp with different flags and dynamically loading the right library variant on the fly based on what we discover, but that support hasn't been merged yet.

I'm not sure yet if we can compile support going all the way back into the 3.7 series, but we'll keep this ticket tracking that.

[datag]

I'd love to see that change. Owner of old GeForce GTX 960M on amd64 Linux here. Version 0.1.18 stopped working while 0.1.17 has been working.

[dhiltgen]

I'd love to see that change. Owner of old GeForce GTX 960M on amd64 Linux here. Version 0.1.18 stopped working while 0.1.17 has been working.

Can you clarify? Was 0.1.17 working on the GPU, or falling back to CPU mode?

Also to clarify, the GTX 960M is a Compute Capability 5.0 card, which we're tracking in a different ticket now #1865

[datag]

I'd love to see that change. Owner of old GeForce GTX 960M on amd64 Linux here. Version 0.1.18 stopped working while 0.1.17 has been working.

Can you clarify? Was 0.1.17 working on the GPU, or falling back to CPU mode?

Also to clarify, the GTX 960M is a Compute Capability 5.0 card, which we're tracking in a different ticket now #1865

You're right, I guess it was falling back to CPU mode, but I'm unsure how to read the logs correctly.

The issue you mentioned seems to be the issue I was having. Version 0.1.19 fixes it. Sorry for the noise and thanks!

[dhiltgen]

but I'm unsure how to read the logs correctly.

At startup the server log will report information about attempting to discover GPU information, and in the case of CUDA cards, will report the compute capability. If we don't detect a supported GPU, we report that we're falling back to CPU mode. In the near future we'll be adding refinements to support multiple variants for a given GPU (and CPU) to try to leverage modern capabilities when detected, but also be able to fallback to a baseline that works for older GPUs/CPUs.

[nejib1]

Hello, same case here, I have Nvidia K80, ollama works only in CPU :(

[sunzh231]

Hi, same case here, I have Nvidia M40, ollama works only in CPU in docker container :(