Confusing error on linux with noexec on /tmp - Error: llama runner process no longer running: 1

[utility-aagrawal]

What is the issue?

I installed ollama on my ubuntu 22.04 machine using the command curl -fsSL https://ollama.com/install.sh | sh

I ran : ollama run llama3 and got this error:
Error: llama runner process no longer running: 1

Can someone help me resolve it?

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.1.32

[dhiltgen]

Can you share your server log?

https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md

[utility-aagrawal]

Thanks for your response, @dhiltgen ! Here are the logs:

journalctl -u ollama
May 02 10:30:33 anurag-Legion-T5-26IRB8 systemd[1]: Started Ollama Service.
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: Couldn't find '/usr/sha>
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: Your new public key is:
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: ssh-ed25519 AAAAC3NzaC1>
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:3>
May 02 10:30:45 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:3>
May 02 10:30:45 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:3>
May 02 10:30:47 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:4>
May 02 10:30:53 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:5>
May 02 10:30:58 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:5>
May 02 10:31:01 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:31:0>
May 02 10:31:14 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:31:1>
May 02 10:31:17 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:31:1>
lines 1-23...skipping...
May 02 10:30:33 anurag-Legion-T5-26IRB8 systemd[1]: Started Ollama Service.
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: Couldn't find '/usr/share/ollama/.ollama/id_ed25519'. Generating new private key.
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: Your new public key is:
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEjXe/9jqrBvZKjWWJmoqWt1Tr6kL2dXrZY0kwVFkBMH
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:33.023-04:00 level=INFO source=images.go:817 msg="total blobs: 0"
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:33.023-04:00 level=INFO source=images.go:824 msg="total unused blobs removed: 0"
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:33.023-04:00 level=INFO source=routes.go:1143 msg="Listening on 127.0.0.1:11434 (version 0.1.32)"
May 02 10:30:33 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:33.024-04:00 level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama566663892/runners
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:34.501-04:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [rocm_v60002 cpu cpu_avx cpu_avx2 cuda_v11]"
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:34.501-04:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:34.501-04:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:34.505-04:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama566663892/runners/cuda_v11/libcudart.so.11.0 /usr/local/cuda/lib64/libcudart.so.12.2.53 /usr/lib/x86_64-linux-gnu/li>
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:34.510-04:00 level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:34.510-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
May 02 10:30:34 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:34.594-04:00 level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.6"
May 02 10:30:45 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:30:45 | 200 | 17.412µs | 127.0.0.1 | HEAD "/"
May 02 10:30:45 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:30:45 | 404 | 61.898µs | 127.0.0.1 | POST "/api/show"
May 02 10:30:47 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:47.690-04:00 level=INFO source=download.go:136 msg="downloading 00e1317cbf74 in 47 100 MB part(s)"
May 02 10:30:53 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:53.691-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 21 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 02 10:30:58 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:30:58.691-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 23 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 02 10:31:01 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:31:01.691-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 11 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 02 10:31:14 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:31:14.691-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 13 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 02 10:31:17 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:31:17.691-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 30 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 02 10:31:18 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:31:18.691-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 41 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 02 10:31:28 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:31:28.691-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 5 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 02 10:33:43 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:33:43 | 404 | 137.724µs | 127.0.0.1 | POST "/api/generate"
May 02 10:33:51 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:33:51 | 404 | 48.228µs | 127.0.0.1 | POST "/api/generate"
May 02 10:36:00 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:36:00 | 200 | 5m14s | 127.0.0.1 | POST "/api/pull"
May 02 10:36:01 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:36:01 | 200 | 13.333µs | 127.0.0.1 | HEAD "/"
May 02 10:36:01 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:36:01 | 404 | 32.331µs | 127.0.0.1 | POST "/api/show"
May 02 10:37:01 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:37:01 | 404 | 72.848µs | 127.0.0.1 | POST "/api/generate"
May 02 10:37:13 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:37:13 | 200 | 1m11s | 127.0.0.1 | POST "/api/pull"
May 02 10:37:26 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:37:26 | 200 | 13.112µs | 127.0.0.1 | HEAD "/"
May 02 10:40:05 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:40:05 | 200 | 2m39s | 127.0.0.1 | POST "/api/pull"
May 02 10:55:46 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:55:46 | 200 | 13.045µs | 127.0.0.1 | HEAD "/"
May 02 10:55:46 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:55:46.983-04:00 level=INFO source=download.go:136 msg="downloading 00e1317cbf74 in 47 100 MB part(s)"
May 02 10:55:55 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:55:55.053-04:00 level=INFO source=download.go:136 msg="downloading 4fa551d4f938 in 1 12 KB part(s)"
May 02 10:55:56 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:55:56.757-04:00 level=INFO source=download.go:136 msg="downloading 8ab4849b038c in 1 254 B part(s)"
May 02 10:55:58 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:55:58.477-04:00 level=INFO source=download.go:136 msg="downloading 577073ffcc6c in 1 110 B part(s)"
May 02 10:56:00 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:00.223-04:00 level=INFO source=download.go:136 msg="downloading ad1518640c43 in 1 483 B part(s)"
May 02 10:56:04 anurag-Legion-T5-26IRB8 ollama[733542]: [GIN] 2024/05/02 - 10:56:04 | 200 | 18.167835427s | 127.0.0.1 | POST "/api/pull"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.182-04:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.182-04:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.185-04:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama566663892/runners/cuda_v11/libcudart.so.11.0 /usr/local/cuda/lib64/libcudart.so.12.2.53 /usr/lib/x86_64-linux-gnu/li>
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.185-04:00 level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.186-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.264-04:00 level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.6"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.274-04:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.274-04:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.275-04:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama566663892/runners/cuda_v11/libcudart.so.11.0 /usr/local/cuda/lib64/libcudart.so.12.2.53 /usr/lib/x86_64-linux-gnu/li>
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.276-04:00 level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.276-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.302-04:00 level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.6"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.312-04:00 level=INFO source=server.go:127 msg="offload to gpu" reallayers=33 layers=33 required="5033.0 MiB" used="5033.0 MiB" available="11127.6 MiB" kv="256.0 MiB" fulloffload="164.0 MiB" partialo>
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.313-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.313-04:00 level=INFO source=server.go:264 msg="starting llama server" cmd="/tmp/ollama566663892/runners/cuda_v11/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-00e1317cbf7>
May 02 10:56:13 anurag-Legion-T5-26IRB8 ollama[733542]: time=2024-05-02T10:56:13.322-04:00 level=INFO source=server.go:389 msg="waiting for llama runner to start responding"

[dhiltgen]

Hmm... waiting for llama runner to start responding as the last message feels like it might be truncated. I would have expected more to follow. However, one possible explanation: /tmp may not allow execution on your setup. Can you try to run /tmp/ollama566663892/runners/cuda_v11/ollama_llama_server --help and see if it runs and gives output, or fails to execute with permission problems? (note: the dir will change each time the server starts)

If that turns out to be the problem, a workaround is documented here: https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#linux-tmp-noexec

[utility-aagrawal]

Thanks @dhiltgen ! That seems like the problem. This is what I see:

/tmp/ollama2753723410/runners/cuda_v11/ollama_llama_server --help
bash: /tmp/ollama2753723410/runners/cuda_v11/ollama_llama_server: Permission denied

How do I know which location user ollama runs as is allowed to write to? I tried a couple of random locations but got the same error. I am sorry but I don't think I understand what the workaround is doing.

[dhiltgen]

Did you try the suggested location in the troublshooting doc OLLAMA_TMPDIR=/usr/share/ollama/ ? This should be writable by the ollama user if you used our standard install script.

What this setting is doing is changing where we write out temporary files, including subprocess executables we run.

[utility-aagrawal]

Thanks @dhiltgen ! I did. Just to clarify, I need to set that as an environment variable using the command "export OLLAMA_TMPDIR=/usr/share/ollama/", right?

[utility-aagrawal]

@dhiltgen , do you have any other suggestions? Is there a way to uninstall everything and try again from scratch?

[utility-aagrawal]

@dhiltgen , just noticed this in server logs:

May 03 14:53:47 anurag-Legion-T5-26IRB8 ollama[771241]: /tmp/ollama1518381580/runners/cuda_v11/ollama_llama_server: /usr/local/cuda/lib64/libcublas.so.11: version `libcublas.so.11' not found (required by /tmp/ollama1518381580/runners/cuda_v11/ollama_llama_server)
May 03 14:53:47 anurag-Legion-T5-26IRB8 ollama[771241]: time=2024-05-03T14:53:47.406-04:00 level=ERROR source=routes.go:120 msg="error loading llama server" error="llama runner process no longer running: 1 "

Does this help? I checked but this file exists:

[dhiltgen]

You should set OLLAMA_TMPDIR for the server - see https://github.com/ollama/ollama/blob/main/docs/faq.md#setting-environment-variables-on-linux for guidance.

If you want to uninstall/re-install see https://github.com/ollama/ollama/blob/main/docs/linux.md#uninstall

For the missing CUDA library, the system should handle this automatically, however you might want to try updating the LD_LIBRARY_PATH for the server to include your cuda library directory from the screenshot to see if that helps.

[utility-aagrawal]

Thanks @dhiltgen !

1. I uninstalled ollama and installed it again using the provided link.
2. I set the OLLAMA_TMPDIR and LD_LIBRARY_PATH for the server.

This is how the file looks after the changes:

Now I am getting a time-out error. Here are server logs:

May 06 11:33:10 anurag-Legion-T5-26IRB8 systemd[1]: Started Ollama Service.
May 06 11:33:10 anurag-Legion-T5-26IRB8 ollama[855890]: Couldn't find '/usr/share/ollama/.ollama/id_ed25519'. Generating new private key.
May 06 11:33:10 anurag-Legion-T5-26IRB8 ollama[855890]: Your new public key is:
May 06 11:33:10 anurag-Legion-T5-26IRB8 ollama[855890]: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG2zyzht7KUX5xyvZayu13WHD7L5+BKCpeAapgk+oa+j
May 06 11:33:10 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:10.631-04:00 level=INFO source=images.go:828 msg="total blobs: 0"
May 06 11:33:10 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:10.631-04:00 level=INFO source=images.go:835 msg="total unused blobs removed: 0"
May 06 11:33:10 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:10.631-04:00 level=INFO source=routes.go:1071 msg="Listening on 127.0.0.1:11434 (version 0.1.33)"
May 06 11:33:10 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:10.640-04:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/usr/share/ollama/runners
May 06 11:33:12 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:12.136-04:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cuda_v11 rocm_v60002 cpu cpu_avx cpu_avx2]"
May 06 11:33:12 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:12.136-04:00 level=INFO source=gpu.go:96 msg="Detecting GPUs"
May 06 11:33:12 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:12.141-04:00 level=INFO source=gpu.go:101 msg="detected GPUs" library=/usr/share/ollama/runners/cuda_v11/libcudart.so.11.0 count=1
May 06 11:33:12 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:12.141-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
May 06 11:33:16 anurag-Legion-T5-26IRB8 ollama[855890]: [GIN] 2024/05/06 - 11:33:16 | 200 | 21.973µs | 127.0.0.1 | HEAD "/"
May 06 11:33:16 anurag-Legion-T5-26IRB8 ollama[855890]: [GIN] 2024/05/06 - 11:33:16 | 404 | 84.248µs | 127.0.0.1 | POST "/api/show"
May 06 11:33:17 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:17.800-04:00 level=INFO source=download.go:136 msg="downloading 00e1317cbf74 in 47 100 MB part(s)"
May 06 11:33:23 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:23.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 16 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:24 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:24.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 46 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:24 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:24.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 22 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:24 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:24.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 12 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:26 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:26.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 18 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:27 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:27.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 34 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:29 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:29.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 3 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:30 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:30.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 7 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:30 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:30.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 43 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:31 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:31.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 39 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:33 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:33.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 13 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:41 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:41.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 30 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:45 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:45.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 35 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:48 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:48.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 26 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:33:56 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:33:56.800-04:00 level=INFO source=download.go:251 msg="00e1317cbf74 part 31 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."
May 06 11:34:24 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:24.617-04:00 level=INFO source=download.go:136 msg="downloading 4fa551d4f938 in 1 12 KB part(s)"
May 06 11:34:26 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:26.330-04:00 level=INFO source=download.go:136 msg="downloading 8ab4849b038c in 1 254 B part(s)"
May 06 11:34:28 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:28.057-04:00 level=INFO source=download.go:136 msg="downloading 577073ffcc6c in 1 110 B part(s)"
May 06 11:34:30 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:30.780-04:00 level=INFO source=download.go:136 msg="downloading ad1518640c43 in 1 483 B part(s)"
May 06 11:34:34 anurag-Legion-T5-26IRB8 ollama[855890]: [GIN] 2024/05/06 - 11:34:34 | 200 | 1m18s | 127.0.0.1 | POST "/api/pull"
May 06 11:34:34 anurag-Legion-T5-26IRB8 ollama[855890]: [GIN] 2024/05/06 - 11:34:34 | 200 | 334.451µs | 127.0.0.1 | POST "/api/show"
May 06 11:34:34 anurag-Legion-T5-26IRB8 ollama[855890]: [GIN] 2024/05/06 - 11:34:34 | 200 | 159.809µs | 127.0.0.1 | POST "/api/show"
May 06 11:34:34 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:34.652-04:00 level=INFO source=gpu.go:96 msg="Detecting GPUs"
May 06 11:34:34 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:34.655-04:00 level=INFO source=gpu.go:101 msg="detected GPUs" library=/usr/share/ollama/runners/cuda_v11/libcudart.so.11.0 count=1
May 06 11:34:34 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:34.655-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
May 06 11:34:35 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:35.115-04:00 level=INFO source=memory.go:152 msg="offload to gpu" layers.real=-1 layers.estimate=33 memory.available="11103.1 MiB" memory.required.full="5033.0 MiB" memory.required.partial="5033.>
May 06 11:34:35 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:35.115-04:00 level=INFO source=memory.go:152 msg="offload to gpu" layers.real=-1 layers.estimate=33 memory.available="11103.1 MiB" memory.required.full="5033.0 MiB" memory.required.partial="5033.>
May 06 11:34:35 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:35.115-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
May 06 11:34:35 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:35.115-04:00 level=INFO source=server.go:289 msg="starting llama server" cmd="/usr/share/ollama/runners/cuda_v11/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-00e1317c>
May 06 11:34:35 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:35.115-04:00 level=INFO source=sched.go:340 msg="loaded runners" count=1
May 06 11:34:35 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:35.115-04:00 level=INFO source=server.go:432 msg="waiting for llama runner to start responding"
May 06 11:34:35 anurag-Legion-T5-26IRB8 ollama[855890]: /usr/share/ollama/runners/cuda_v11/ollama_llama_server: /usr/local/cuda/lib64/libcublas.so.11: version `libcublas.so.11' not found (required by /usr/share/ollama/runners/cuda_v11/ollama_llama_server)
May 06 11:34:46 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:46.522-04:00 level=INFO source=server.go:437 msg="context expired before server started"
May 06 11:34:46 anurag-Legion-T5-26IRB8 ollama[855890]: time=2024-05-06T11:34:46.522-04:00 level=ERROR source=sched.go:346 msg="error loading llama server" error="timed out waiting for llama runner to start: context canceled"

Can you please advise what might be wrong? Thanks for your assistance!

[dhiltgen]

Can you please advise what might be wrong? Thanks for your assistance!

It looks like we may have a bug in wiring up the LD_LIBRARY_PATH properly when OLLAMA_TMPDIR is set. Investigating...

[dhiltgen]

@utility-aagrawal can you try 0.1.34?

[utility-aagrawal]

Thanks for your response, @dhiltgen !

I was able to make it work. For me, the issue wasn't ollama related. I have cuda12 on my machine but I had both libcublas.so.11 and libcublas.so.12. I remember creating a symbolic link from 12 to 11 for some other program to run. As soon as I removed libcublas.so.11, ollama worked.

I can confirm that it still works with the latest version 0.1.34. Thanks again for your help!

[utility-aagrawal]

Closing this issue. Thanks!