Ollama fails to start in CPU only mode #3526

vishnu-dev · 2024-04-07T16:26:08Z

What is the issue?

Ollama fails to start properly when using in a system with only CPU mode. This happened after I upgraded to latest version i.e. 0.1.30 using the curl command as in the docs. I had to downgrade by uninstalling and installing specific version as in docs, but still the problem persists. Tried with docker after this mishap, still doesn't work.

Following are the logs using journalctl:

Apr 07 14:38:53 ollama[1344954]: time=2024-04-07T14:38:53.025Z level=INFO source=images.go:706 msg="total blobs: 0"
Apr 07 14:38:53 ollama[1344954]: time=2024-04-07T14:38:53.025Z level=INFO source=images.go:713 msg="total unused blobs removed: 0"
Apr 07 14:38:53 ollama[1344954]: time=2024-04-07T14:38:53.025Z level=INFO source=routes.go:1014 msg="Listening on 127.0.0.1:11434 (version 0.1.25)"
Apr 07 14:38:53 ollama[1344954]: time=2024-04-07T14:38:53.026Z level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
Apr 07 14:38:55 ollama[1344954]: time=2024-04-07T14:38:55.998Z level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [rocm_v6 cpu_avx2 cpu_avx cuda_v11 cpu rocm>
Apr 07 14:38:55 ollama[1344954]: time=2024-04-07T14:38:55.998Z level=INFO source=gpu.go:94 msg="Detecting GPU type"
Apr 07 14:38:55 ollama[1344954]: time=2024-04-07T14:38:55.998Z level=INFO source=gpu.go:262 msg="Searching for GPU management library libnvidia-ml.so"
Apr 07 14:38:56 ollama[1344954]: time=2024-04-07T14:38:56.000Z level=INFO source=gpu.go:308 msg="Discovered GPU libraries: []"
Apr 07 14:38:56 ollama[1344954]: time=2024-04-07T14:38:56.000Z level=INFO source=gpu.go:262 msg="Searching for GPU management library librocm_smi64.so"
Apr 07 14:38:56 ollama[1344954]: time=2024-04-07T14:38:56.000Z level=INFO source=gpu.go:308 msg="Discovered GPU libraries: []"
Apr 07 14:38:56 ollama[1344954]: time=2024-04-07T14:38:56.000Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
Apr 07 14:38:56 ollama[1344954]: time=2024-04-07T14:38:56.000Z level=INFO source=routes.go:1037 msg="no GPU detected"

All commands fail, for example:

$ ollama -v
Warning: could not connect to a running Ollama instance
Warning: client version is 0.1.25

$ ollama list
Error: something went wrong, please see the ollama server logs for details

What did you expect to see?

Expected normal functioning and launch. Updates shouldn't affect the system.

Steps to reproduce

All steps on a CPU only configuration:

Regular working version ollama - v --> 0.1.25
Update ollama curl -fsSL https://ollama.com/install.sh | sh
Check version now ollama -v --> 0.1.30 --> Doesn't start serving

Are there any recent changes that introduced the issue?

Update ollama curl -fsSL https://ollama.com/install.sh | sh

OS

Linux

Architecture

x86

Platform

No response

Ollama version

0.1.25

GPU

No response

GPU info

No response

CPU

Intel

Other software

CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Supports:

Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_goo
                                    d nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch in
                                    vpcid_single tpr_shadow vnmi ept vpid ept_ad fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd
                                    avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_vnni arch_capabilities

The text was updated successfully, but these errors were encountered:

navr32 · 2024-04-08T10:23:57Z

Please provide the full log of ollama serve !

vishnu-dev · 2024-04-08T10:35:56Z

Please provide the full log of ollama serve !

This is the full log of Ollama from journalctl. It doesn't proceed after this, that's the problem.

Tamaya31 · 2024-04-10T07:56:20Z

I am facing the exact same error. The only thing we did is try to delete / create custom models from model files and the error started to appear. No additional logs about the problem is printed.

vishnu-dev · 2024-04-10T11:54:48Z

@Tamaya31 Were you able to solve this?

vishnu-dev · 2024-04-15T06:34:37Z

@navr32 I enabled OLLAMA_DEBUG and got some additional logs:

time=2024-04-15T06:33:25.816Z level=INFO source=images.go:804 msg="total blobs: 0"
time=2024-04-15T06:33:25.816Z level=INFO source=images.go:811 msg="total unused blobs removed: 0"
time=2024-04-15T06:33:25.816Z level=INFO source=routes.go:1118 msg="Listening on [::]:11434 (version 0.1.31)"
time=2024-04-15T06:33:25.817Z level=INFO source=payload_common.go:113 msg="Extracting dynamic libraries to /tmp/ollama727024826/runners ..."
time=2024-04-15T06:33:28.909Z level=INFO source=payload_common.go:140 msg="Dynamic LLM libraries [rocm_v60000 cpu cpu_avx2 cuda_v11 cpu_avx]"
time=2024-04-15T06:33:28.909Z level=DEBUG source=payload_common.go:141 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-04-15T06:33:28.909Z level=INFO source=gpu.go:115 msg="Detecting GPU type"
time=2024-04-15T06:33:28.909Z level=INFO source=gpu.go:265 msg="Searching for GPU management library libcudart.so*"
time=2024-04-15T06:33:28.909Z level=DEBUG source=gpu.go:283 msg="gpu management search paths: [/tmp/ollama727024826/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /home/MPCDAdmin/libcudart.so**]"
time=2024-04-15T06:33:28.910Z level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [/tmp/ollama727024826/runners/cuda_v11/libcudart.so.11.0]"
wiring cudart library functions in /tmp/ollama727024826/runners/cuda_v11/libcudart.so.11.0
dlsym: cudaSetDevice
dlsym: cudaDeviceSynchronize
dlsym: cudaDeviceReset
dlsym: cudaMemGetInfo
dlsym: cudaGetDeviceCount
dlsym: cudaDeviceGetAttribute
dlsym: cudaDriverGetVersion
cudaSetDevice err: 35
time=2024-04-15T06:33:28.911Z level=INFO source=gpu.go:340 msg="Unable to load cudart CUDA management library /tmp/ollama727024826/runners/cuda_v11/libcudart.so.11.0: cudart init failure: 35"
time=2024-04-15T06:33:28.911Z level=INFO source=gpu.go:265 msg="Searching for GPU management library libnvidia-ml.so"
time=2024-04-15T06:33:28.911Z level=DEBUG source=gpu.go:283 msg="gpu management search paths: [/usr/local/cuda/lib64/libnvidia-ml.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-ml.so* /usr/lib/x86_64-linux-gnu/libnvidia-ml.so* /usr/lib/wsl/lib/libnvidia-ml.so* /usr/lib/wsl/drivers/*/libnvidia-ml.so* /opt/cuda/lib64/libnvidia-ml.so* /usr/lib*/libnvidia-ml.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libnvidia-ml.so* /usr/lib/aarch64-linux-gnu/libnvidia-ml.so* /usr/local/lib*/libnvidia-ml.so* /opt/cuda/targets/x86_64-linux/lib/stubs/libnvidia-ml.so* /home/MPCDAdmin/libnvidia-ml.so*]"
time=2024-04-15T06:33:28.913Z level=INFO source=gpu.go:311 msg="Discovered GPU libraries: []"
time=2024-04-15T06:33:28.913Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-15T06:33:28.913Z level=DEBUG source=amd_linux.go:263 msg="amdgpu driver not detected /sys/module/amdgpu"
time=2024-04-15T06:33:28.913Z level=INFO source=routes.go:1141 msg="no GPU detected"

vishnu-dev · 2024-04-15T07:16:47Z

Was able to find a workaround for this issue. Hope it helps someone in need.

Since none of the CLI commands work in this case Eg: ollama pull phi, we use the API as an alternative to interact with ollama. So, we do POST /api/pull for {"name": "phi"}. You can now interact with ollama using the API for chat.

Tamaya31 · 2024-04-17T05:47:04Z

@Tamaya31 Were you able to solve this?

Sadly no. I went with the docker image as an alternative for now.

In the container, `OLLAMA_HOST` is set by default to `0.0.0.0` (ref: [Dockerfile#L137]), which is fine when starting the server. However, as a client, it is necessary to use `127.0.0.1` or `localhost` for requests. fix: ollama#3521 ollama#1337 maybe fix: ollama#3526 [Dockerfile#L137]: https://github.com/ollama/ollama/blob/7e432cdfac51583459e7bfa8fdd485c74a6597e7/Dockerfile#L137 Signed-off-by: Kevin Cui <bh@bugs.cc>

In the container, `OLLAMA_HOST` is set by default to `0.0.0.0` (ref: [Dockerfile#L137]), which is fine when starting the server. However, as a client, it is must to use `127.0.0.1` or `localhost` for requests. fix: ollama#3521 ollama#1337 maybe fix: ollama#3526 [Dockerfile#L137]: https://github.com/ollama/ollama/blob/7e432cdfac51583459e7bfa8fdd485c74a6597e7/Dockerfile#L137 Signed-off-by: Kevin Cui <bh@bugs.cc>

dhiltgen · 2024-05-05T00:42:14Z

Are you still seeing this failure with the latest version 0.1.33? If so, can you share an updated server log? The log output you shared above ending with these 2 lines looks like a typical startup, with the server now waiting for requests from the client

Apr 07 14:38:56 ollama[1344954]: time=2024-04-07T14:38:56.000Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
Apr 07 14:38:56 ollama[1344954]: time=2024-04-07T14:38:56.000Z level=INFO source=routes.go:1037 msg="no GPU detected"

pdevine · 2024-05-18T04:12:42Z

Hey @vishnu-dev , I think everything should be working correctly. It looks like you were connecting to it on the wrong port? If you're still seeing this, please share the server logs as @dhiltgen mentioned and we can reopen.

vishnu-dev added bug Something isn't working needs-triage labels Apr 7, 2024

vishnu-dev mentioned this issue Apr 10, 2024

Run ollama list/pull got an error #3521

Closed

jmorganca removed the needs-triage label Apr 19, 2024

BlackHole1 mentioned this issue Apr 29, 2024

fix(cli): unable to use CLI within the container #4023

Closed

dhiltgen self-assigned this May 5, 2024

pdevine closed this as completed May 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ollama fails to start in CPU only mode #3526

Ollama fails to start in CPU only mode #3526

vishnu-dev commented Apr 7, 2024 •

edited

navr32 commented Apr 8, 2024

vishnu-dev commented Apr 8, 2024

Tamaya31 commented Apr 10, 2024

vishnu-dev commented Apr 10, 2024

vishnu-dev commented Apr 15, 2024 •

edited

vishnu-dev commented Apr 15, 2024 •

edited

Tamaya31 commented Apr 17, 2024

dhiltgen commented May 5, 2024

pdevine commented May 18, 2024

Ollama fails to start in CPU only mode #3526

Ollama fails to start in CPU only mode #3526

Comments

vishnu-dev commented Apr 7, 2024 • edited

What is the issue?

What did you expect to see?

Steps to reproduce

Are there any recent changes that introduced the issue?

OS

Architecture

Platform

Ollama version

GPU

GPU info

CPU

Other software

navr32 commented Apr 8, 2024

vishnu-dev commented Apr 8, 2024

Tamaya31 commented Apr 10, 2024

vishnu-dev commented Apr 10, 2024

vishnu-dev commented Apr 15, 2024 • edited

vishnu-dev commented Apr 15, 2024 • edited

Tamaya31 commented Apr 17, 2024

dhiltgen commented May 5, 2024

pdevine commented May 18, 2024

vishnu-dev commented Apr 7, 2024 •

edited

vishnu-dev commented Apr 15, 2024 •

edited

vishnu-dev commented Apr 15, 2024 •

edited