Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: Post "http://127.0.0.1:11434/api/chat": read tcp 127.0.0.1:59108->127.0.0.1:11434: wsarecv: An existing connection was forcibly closed by the remote host. #3593

Closed
liquorLiu opened this issue Apr 11, 2024 · 13 comments
Closed

Error: Post "http://127.0.0.1:11434/api/chat": read tcp 127.0.0.1:59108->127.0.0.1:11434: wsarecv: An existing connection was forcibly closed by the remote host. #3593

liquorLiu opened this issue Apr 11, 2024 · 13 comments
Assignees
@dhiltgen
Labels
bug Something isn't working windows

Comments

@liquorLiu
Copy link

here is my app.log:
time=2024-04-11T19:53:53.911+08:00 level=INFO source=logging.go:45 msg="ollama app started"
time=2024-04-11T19:53:53.940+08:00 level=DEBUG source=lifecycle.go:32 msg="starting callback loop"
time=2024-04-11T19:53:53.940+08:00 level=DEBUG source=store.go:61 msg="loaded existing store C:\Users\17900\AppData\Local\Ollama\config.json - ID: ef96d596-af07-43a7-97e2-f6af540c5021"
time=2024-04-11T19:53:53.940+08:00 level=DEBUG source=lifecycle.go:66 msg="Not first time, skipping first run notification"
time=2024-04-11T19:53:53.941+08:00 level=DEBUG source=server.go:134 msg="heartbeat from server: Head "http://127.0.0.1:11434/\": dial tcp 127.0.0.1:11434: connectex: No connection could be made because the target machine actively refused it."
time=2024-04-11T19:53:53.941+08:00 level=INFO source=server.go:135 msg="unable to connect to server"
time=2024-04-11T19:53:53.947+08:00 level=INFO source=server.go:91 msg="started ollama server with pid 7836"
time=2024-04-11T19:53:53.947+08:00 level=INFO source=server.go:93 msg="ollama server logs C:\Users\17900\AppData\Local\Ollama\server.log"
time=2024-04-11T19:53:53.947+08:00 level=DEBUG source=eventloop.go:24 msg="starting event handling loop"
time=2024-04-11T19:53:56.952+08:00 level=DEBUG source=updater.go:73 msg="checking for available update" requestURL="https://ollama.com/api/update?arch=amd64&nonce=4ubr2F5IgZhBzSseU_H9iQ&os=windows&ts=1712836436&version=0.1.31"
time=2024-04-11T19:53:58.123+08:00 level=DEBUG source=updater.go:82 msg="check update response 204 (current version is up to date)"
time=2024-04-11T19:54:06.001+08:00 level=WARN source=server.go:113 msg="server crash 1 - exit code 3221226505 - respawning"
time=2024-04-11T19:54:06.509+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:54:07.516+08:00 level=WARN source=server.go:113 msg="server crash 2 - exit code 3221226505 - respawning"
time=2024-04-11T19:54:08.022+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:54:10.031+08:00 level=WARN source=server.go:113 msg="server crash 3 - exit code 3221226505 - respawning"
time=2024-04-11T19:54:10.536+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:54:12.217+08:00 level=DEBUG source=logging_windows.go:12 msg="viewing logs with start C:\Users\17900\AppData\Local\Ollama"
time=2024-04-11T19:54:13.540+08:00 level=WARN source=server.go:113 msg="server crash 4 - exit code 3221226505 - respawning"
time=2024-04-11T19:54:14.052+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:54:18.066+08:00 level=WARN source=server.go:113 msg="server crash 5 - exit code 3221226505 - respawning"
time=2024-04-11T19:54:18.573+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:54:23.575+08:00 level=WARN source=server.go:113 msg="server crash 6 - exit code 3221226505 - respawning"
time=2024-04-11T19:54:24.081+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:54:30.085+08:00 level=WARN source=server.go:113 msg="server crash 7 - exit code 3221226505 - respawning"
time=2024-04-11T19:54:30.592+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:54:37.604+08:00 level=WARN source=server.go:113 msg="server crash 8 - exit code 3221226505 - respawning"
time=2024-04-11T19:54:38.106+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:54:46.117+08:00 level=WARN source=server.go:113 msg="server crash 9 - exit code 3221226505 - respawning"
time=2024-04-11T19:54:46.626+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:54:55.632+08:00 level=WARN source=server.go:113 msg="server crash 10 - exit code 3221226505 - respawning"
time=2024-04-11T19:54:56.140+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:55:06.150+08:00 level=WARN source=server.go:113 msg="server crash 11 - exit code 3221226505 - respawning"
time=2024-04-11T19:55:06.655+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:55:17.667+08:00 level=WARN source=server.go:113 msg="server crash 12 - exit code 3221226505 - respawning"
time=2024-04-11T19:55:18.177+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:55:30.185+08:00 level=WARN source=server.go:113 msg="server crash 13 - exit code 3221226505 - respawning"
time=2024-04-11T19:55:30.693+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:55:43.694+08:00 level=WARN source=server.go:113 msg="server crash 14 - exit code 3221226505 - respawning"
time=2024-04-11T19:55:44.205+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:55:58.210+08:00 level=WARN source=server.go:113 msg="server crash 15 - exit code 3221226505 - respawning"
time=2024-04-11T19:55:58.718+08:00 level=ERROR source=server.go:116 msg="failed to restart server exec: already started"
time=2024-04-11T19:56:06.457+08:00 level=DEBUG source=eventloop.go:148 msg="unmanaged app message, lParm: 0x204"
time=2024-04-11T19:56:07.035+08:00 level=DEBUG source=logging_windows.go:12 msg="viewing logs with start C:\Users\17900\AppData\Local\Ollama"

how to fix it?thanks
@ethanlchristensen
Copy link

ethanlchristensen commented Apr 15, 2024 •
edited

I've encountered similar problems on windows after I shut my PC down and start it up again the next day. Starting it up again the next day, I can see Ollama running, but running any command throws that error.
image
image

The only way I have been able to solve the issues is to kill the process that is actively holding the port and restarting Ollama.

image

@dhiltgen dhiltgen self-assigned this Apr 16, 2024
@dhiltgen
Copy link
Collaborator

Unfortunately it looks like your server is crashing. Can you share your server.log so we can see why?

@dhiltgen dhiltgen added the bug Something isn't working label Apr 16, 2024
@liquorLiu
Copy link
Author

Unfortunately it looks like your server is crashing. Can you share your server.log so we can see why?

time=2024-04-16T09:51:41.678+08:00 level=INFO source=gpu.go:115 msg="Detecting GPU type"
time=2024-04-16T09:51:41.678+08:00 level=INFO source=gpu.go:265 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-16T09:51:41.678+08:00 level=DEBUG source=gpu.go:283 msg="gpu management search paths: ...
time=2024-04-16T09:51:41.688+08:00 level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [C:\Users\17900\AppData\Local\Programs\Ollama\cudart64_110.dll]"

my GPU break,i just delete it,then everything is ok.

@liquorLiu
Copy link
Author

I've encountered similar problems on windows after I shut my PC down and start it up again the next day. Starting it up again the next day, I can see Ollama running, but running any command throws that error. image image

The only way I have been able to solve the issues is to kill the process that is actively holding the port and restarting Ollama.

image

set OLLAMA_DEBUG=1,then look your sever.log and app.log,i think we are not same mistake.show it,maybe someone can help you

@dhiltgen
Copy link
Collaborator

dhiltgen commented Apr 16, 2024 •
edited

@liquorLiu that log doesn't seem to contain a crash or any error messages. Let's try a different approach to try to understand what's going wrong.

Please Quit the tray app, and in a powershell terminal run:

$env:OLLAMA_DEBUG="1"
ollama serve 2>&1 | % ToString | Tee-Object server.log

Assuming that doesn't immediately crash and give you back a prompt, in another terminal window run

ollama run llama2 hello

After it completes, or errors out, in the first window hit <ctrl>-c and attach the server.log to this issue so I can take a look at a full run.

If the ollama serve does immediately crash, share what you see in the terminal, and server.log if it's not empty.

@timfduffy
Copy link

Hi @dhiltgen, I think I may be experiencing the same issue as @liquorLiu, based on the fact that I experienced the same error message in the post title, and well as the same error in app.log and no errors in server.log. Before I applied the steps in your most recent post, here is what my log files looked like:

app.log:

time=2024-04-19T12:43:51.243-07:00 level=INFO source=logging.go:45 msg="ollama app started"
time=2024-04-19T12:43:51.259-07:00 level=INFO source=server.go:167 msg="unable to connect to server"
time=2024-04-19T12:43:51.262-07:00 level=INFO source=server.go:123 msg="started ollama server with pid 2520"
time=2024-04-19T12:43:51.262-07:00 level=INFO source=server.go:125 msg="ollama server logs C:\\Users\\timfd\\AppData\\Local\\Ollama\\server.log"
time=2024-04-19T12:43:53.435-07:00 level=WARN source=server.go:145 msg="server crash 1 - exit code 3221226505 - respawning"
time=2024-04-19T12:43:53.949-07:00 level=ERROR source=server.go:148 msg="failed to restart server exec: already started"
time=2024-04-19T12:43:54.963-07:00 level=WARN source=server.go:145 msg="server crash 2 - exit code 3221226505 - respawning"
time=2024-04-19T12:43:55.475-07:00 level=ERROR source=server.go:148 msg="failed to restart server exec: already started"
time=2024-04-19T12:43:57.486-07:00 level=WARN source=server.go:145 msg="server crash 3 - exit code 3221226505 - respawning"

server.log:

time=2024-04-19T12:43:51.304-07:00 level=INFO source=images.go:817 msg="total blobs: 7"
time=2024-04-19T12:43:51.304-07:00 level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-04-19T12:43:51.305-07:00 level=INFO source=routes.go:1143 msg="Listening on 127.0.0.1:11434 (version 0.1.32)"
time=2024-04-19T12:43:51.306-07:00 level=INFO source=payload.go:28 msg="extracting embedded files" dir=C:\Users\timfd\AppData\Local\Temp\ollama3627894243\runners
time=2024-04-19T12:43:51.546-07:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11.3 rocm_v5.7]"
[GIN] 2024/04/19 - 12:43:51 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2024/04/19 - 12:43:51 | 200 |       535.8µs |       127.0.0.1 | POST     "/api/show"
time=2024-04-19T12:43:53.344-07:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-19T12:43:53.344-07:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-19T12:43:53.349-07:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [C:\\Users\\timfd\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_65.dll]"

I ran the two lines you provided for PowerShell above, it didn't crash though I'm not sure it ran correctly, it added this to my server.log:

failed to get console mode for stdout: The handle is invalid.
failed to get console mode for stderr: The handle is invalid.
time=2024-04-19T12:45:46.523-07:00 level=INFO source=images.go:817 msg="total blobs: 7"
time=2024-04-19T12:45:46.524-07:00 level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-04-19T12:45:46.524-07:00 level=INFO source=routes.go:1143 msg="Listening on 127.0.0.1:11434 (version 0.1.32)"
time=2024-04-19T12:45:46.534-07:00 level=INFO source=payload.go:28 msg="extracting embedded files" dir=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/windows/amd64/cpu/bin/llama.dll.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/windows/amd64/cpu/bin/ollama_llama_server.exe.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/windows/amd64/cpu_avx/bin/llama.dll.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/windows/amd64/cpu_avx/bin/ollama_llama_server.exe.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/windows/amd64/cpu_avx2/bin/llama.dll.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/windows/amd64/cpu_avx2/bin/ollama_llama_server.exe.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11.3 file=build/windows/amd64/cuda_v11.3/bin/llama.dll.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11.3 file=build/windows/amd64/cuda_v11.3/bin/ollama_llama_server.exe.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v5.7 file=build/windows/amd64/rocm_v5.7/bin/llama.dll.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v5.7 file=build/windows/amd64/rocm_v5.7/bin/ollama_llama_server.exe.gz
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners\cpu
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners\cpu_avx
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners\cpu_avx2
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners\cuda_v11.3
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners\rocm_v5.7
time=2024-04-19T12:45:46.772-07:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu_avx cpu_avx2 cuda_v11.3 rocm_v5.7 cpu]"
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"

After that I ran the command ollama run llama3 hello, this is what was added to server.log:

[GIN] 2024/04/19 - 12:49:13 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2024/04/19 - 12:49:13 | 200 |      1.0441ms |       127.0.0.1 | POST     "/api/show"
time=2024-04-19T12:49:13.479-07:00 level=DEBUG source=gguf.go:57 msg="model = &llm.gguf{containerGGUF:(*llm.containerGGUF)(0xc000471180), kv:llm.KV{}, tensors:[]*llm.Tensor(nil), parameters:0x0}"
time=2024-04-19T12:49:15.061-07:00 level=DEBUG source=gguf.go:193 msg="general.architecture = llama"
time=2024-04-19T12:49:15.065-07:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-19T12:49:15.065-07:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-19T12:49:15.065-07:00 level=DEBUG source=gpu.go:286 msg="gpu management search paths: [C:\\Users\\timfd\\AppData\\Local\\Programs\\Ollama\\cudart64_*.dll c:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v*\\bin\\cudart64_*.dll C:\\Program Files (x86)\\Common Files\\Oracle\\Java\\javapath\\cudart64_*.dll* C:\\Program Files\\Oculus\\Support\\oculus-runtime\\cudart64_*.dll* C:\\Windows\\system32\\cudart64_*.dll* C:\\Windows\\cudart64_*.dll* C:\\Windows\\System32\\Wbem\\cudart64_*.dll* C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\cudart64_*.dll* C:\\Windows\\System32\\OpenSSH\\cudart64_*.dll* C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_*.dll* C:\\Program Files\\NVIDIA Corporation\\NVIDIA NvDLISR\\cudart64_*.dll* C:\\Program Files\\dotnet\\cudart64_*.dll* C:\\Program Files (x86)\\AOMEI\\AOMEI Backupper\\7.3.1\\cudart64_*.dll* C:\\Program Files\\Mullvad VPN\\resources\\cudart64_*.dll* C:\\Program Files\\Docker\\Docker\\resources\\bin\\cudart64_*.dll* C:\\Program Files\\Python311\\Scripts\\cudart64_*.dll* C:\\Users\\timfd\\AppData\\Local\\Microsoft\\WindowsApps\\cudart64_*.dll* C:\\Users\\timfd\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\cudart64_*.dll* C:\\Program Files\\Python311\\cudart64_*.dll* C:\\Users\\timfd\\AppData\\Local\\Ollama\\cudart64_*.dll* C:\\Users\\timfd\\AppData\\Local\\Programs\\Ollama\\cudart64_*.dll*]"
time=2024-04-19T12:49:15.069-07:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [C:\\Users\\timfd\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_65.dll]"

Please let me know if there is any other information that would be helpful to provide, or if I should post this somewhere else, I'm quite new to all of this.

@liquorLiu
Copy link
Author

Hi @dhiltgen, I think I may be experiencing the same issue as @liquorLiu, based on the fact that I experienced the same error message in the post title, and well as the same error in app.log and no errors in server.log. Before I applied the steps in your most recent post, here is what my log files looked like:

app.log:

time=2024-04-19T12:43:51.243-07:00 level=INFO source=logging.go:45 msg="ollama app started"
time=2024-04-19T12:43:51.259-07:00 level=INFO source=server.go:167 msg="unable to connect to server"
time=2024-04-19T12:43:51.262-07:00 level=INFO source=server.go:123 msg="started ollama server with pid 2520"
time=2024-04-19T12:43:51.262-07:00 level=INFO source=server.go:125 msg="ollama server logs C:\\Users\\timfd\\AppData\\Local\\Ollama\\server.log"
time=2024-04-19T12:43:53.435-07:00 level=WARN source=server.go:145 msg="server crash 1 - exit code 3221226505 - respawning"
time=2024-04-19T12:43:53.949-07:00 level=ERROR source=server.go:148 msg="failed to restart server exec: already started"
time=2024-04-19T12:43:54.963-07:00 level=WARN source=server.go:145 msg="server crash 2 - exit code 3221226505 - respawning"
time=2024-04-19T12:43:55.475-07:00 level=ERROR source=server.go:148 msg="failed to restart server exec: already started"
time=2024-04-19T12:43:57.486-07:00 level=WARN source=server.go:145 msg="server crash 3 - exit code 3221226505 - respawning"

server.log:

time=2024-04-19T12:43:51.304-07:00 level=INFO source=images.go:817 msg="total blobs: 7"
time=2024-04-19T12:43:51.304-07:00 level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-04-19T12:43:51.305-07:00 level=INFO source=routes.go:1143 msg="Listening on 127.0.0.1:11434 (version 0.1.32)"
time=2024-04-19T12:43:51.306-07:00 level=INFO source=payload.go:28 msg="extracting embedded files" dir=C:\Users\timfd\AppData\Local\Temp\ollama3627894243\runners
time=2024-04-19T12:43:51.546-07:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11.3 rocm_v5.7]"
[GIN] 2024/04/19 - 12:43:51 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2024/04/19 - 12:43:51 | 200 |       535.8µs |       127.0.0.1 | POST     "/api/show"
time=2024-04-19T12:43:53.344-07:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-19T12:43:53.344-07:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-19T12:43:53.349-07:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [C:\\Users\\timfd\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_65.dll]"

I ran the two lines you provided for PowerShell above, it didn't crash though I'm not sure it ran correctly, it added this to my server.log:

failed to get console mode for stdout: The handle is invalid.
failed to get console mode for stderr: The handle is invalid.
time=2024-04-19T12:45:46.523-07:00 level=INFO source=images.go:817 msg="total blobs: 7"
time=2024-04-19T12:45:46.524-07:00 level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-04-19T12:45:46.524-07:00 level=INFO source=routes.go:1143 msg="Listening on 127.0.0.1:11434 (version 0.1.32)"
time=2024-04-19T12:45:46.534-07:00 level=INFO source=payload.go:28 msg="extracting embedded files" dir=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/windows/amd64/cpu/bin/llama.dll.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/windows/amd64/cpu/bin/ollama_llama_server.exe.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/windows/amd64/cpu_avx/bin/llama.dll.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/windows/amd64/cpu_avx/bin/ollama_llama_server.exe.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/windows/amd64/cpu_avx2/bin/llama.dll.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/windows/amd64/cpu_avx2/bin/ollama_llama_server.exe.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11.3 file=build/windows/amd64/cuda_v11.3/bin/llama.dll.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11.3 file=build/windows/amd64/cuda_v11.3/bin/ollama_llama_server.exe.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v5.7 file=build/windows/amd64/rocm_v5.7/bin/llama.dll.gz
time=2024-04-19T12:45:46.534-07:00 level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v5.7 file=build/windows/amd64/rocm_v5.7/bin/ollama_llama_server.exe.gz
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners\cpu
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners\cpu_avx
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners\cpu_avx2
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners\cuda_v11.3
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\timfd\AppData\Local\Temp\ollama1521772082\runners\rocm_v5.7
time=2024-04-19T12:45:46.772-07:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu_avx cpu_avx2 cuda_v11.3 rocm_v5.7 cpu]"
time=2024-04-19T12:45:46.772-07:00 level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"

After that I ran the command ollama run llama3 hello, this is what was added to server.log:

[GIN] 2024/04/19 - 12:49:13 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2024/04/19 - 12:49:13 | 200 |      1.0441ms |       127.0.0.1 | POST     "/api/show"
time=2024-04-19T12:49:13.479-07:00 level=DEBUG source=gguf.go:57 msg="model = &llm.gguf{containerGGUF:(*llm.containerGGUF)(0xc000471180), kv:llm.KV{}, tensors:[]*llm.Tensor(nil), parameters:0x0}"
time=2024-04-19T12:49:15.061-07:00 level=DEBUG source=gguf.go:193 msg="general.architecture = llama"
time=2024-04-19T12:49:15.065-07:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-19T12:49:15.065-07:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-19T12:49:15.065-07:00 level=DEBUG source=gpu.go:286 msg="gpu management search paths: [C:\\Users\\timfd\\AppData\\Local\\Programs\\Ollama\\cudart64_*.dll c:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v*\\bin\\cudart64_*.dll C:\\Program Files (x86)\\Common Files\\Oracle\\Java\\javapath\\cudart64_*.dll* C:\\Program Files\\Oculus\\Support\\oculus-runtime\\cudart64_*.dll* C:\\Windows\\system32\\cudart64_*.dll* C:\\Windows\\cudart64_*.dll* C:\\Windows\\System32\\Wbem\\cudart64_*.dll* C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\cudart64_*.dll* C:\\Windows\\System32\\OpenSSH\\cudart64_*.dll* C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_*.dll* C:\\Program Files\\NVIDIA Corporation\\NVIDIA NvDLISR\\cudart64_*.dll* C:\\Program Files\\dotnet\\cudart64_*.dll* C:\\Program Files (x86)\\AOMEI\\AOMEI Backupper\\7.3.1\\cudart64_*.dll* C:\\Program Files\\Mullvad VPN\\resources\\cudart64_*.dll* C:\\Program Files\\Docker\\Docker\\resources\\bin\\cudart64_*.dll* C:\\Program Files\\Python311\\Scripts\\cudart64_*.dll* C:\\Users\\timfd\\AppData\\Local\\Microsoft\\WindowsApps\\cudart64_*.dll* C:\\Users\\timfd\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\cudart64_*.dll* C:\\Program Files\\Python311\\cudart64_*.dll* C:\\Users\\timfd\\AppData\\Local\\Ollama\\cudart64_*.dll* C:\\Users\\timfd\\AppData\\Local\\Programs\\Ollama\\cudart64_*.dll*]"
time=2024-04-19T12:49:15.069-07:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [C:\\Users\\timfd\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_65.dll]"

Please let me know if there is any other information that would be helpful to provide, or if I should post this somewhere else, I'm quite new to all of this.

i just remove C:\Users\timfd\AppData\Local\Programs\Ollama\cudart64_110.dll,then everythings is ok,
maybe your gpu break down like me.just use cpu

@NAME0x0
Copy link

NAME0x0 commented Apr 20, 2024 •
edited

I have an rx 6600 xt gpu and I tried the HSA_override_gfx environment variable to get ollama use my gpu since my cpu is old and incapable of running models with 7b parameters despite having enough ram and storage. i even tried adding the rocm files from the preview build as provided in the repo https://github.com/brknsoul/ROCmLibs. However, even if I remove the hsa_override variable i keep getting this error:
Error: llama runner process no longer running: 3221226505
after using the powershell command that was suggested, this is the error log that comes up:
failed to get console mode for stdout: The handle is invalid. failed to get console mode for stderr: The handle is invalid. time=2024-04-20T22:04:24.000+02:00 level=INFO source=images.go:817 msg="total blobs: 180" time=2024-04-20T22:04:24.080+02:00 level=INFO source=images.go:824 msg="total unused blobs removed: 0" time=2024-04-20T22:04:24.132+02:00 level=INFO source=routes.go:1143 msg="Listening on 127.0.0.1:11434 (version 0.1.32)" time=2024-04-20T22:04:24.147+02:00 level=INFO source=payload.go:28 msg="extracting embedded files" dir=C:\Users\name\AppData\Local\Temp\ollama522826981\runners time=2024-04-20T22:04:24.148+02:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/windows/amd64/cpu/bin/llama.dll.gz time=2024-04-20T22:04:24.149+02:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/windows/amd64/cpu/bin/ollama_llama_server.exe.gz time=2024-04-20T22:04:24.149+02:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/windows/amd64/cpu_avx/bin/llama.dll.gz time=2024-04-20T22:04:24.149+02:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/windows/amd64/cpu_avx/bin/ollama_llama_server.exe.gz time=2024-04-20T22:04:24.149+02:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/windows/amd64/cpu_avx2/bin/llama.dll.gz time=2024-04-20T22:04:24.149+02:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/windows/amd64/cpu_avx2/bin/ollama_llama_server.exe.gz time=2024-04-20T22:04:24.149+02:00 level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11.3 file=build/windows/amd64/cuda_v11.3/bin/llama.dll.gz time=2024-04-20T22:04:24.149+02:00 level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11.3 file=build/windows/amd64/cuda_v11.3/bin/ollama_llama_server.exe.gz time=2024-04-20T22:04:24.150+02:00 level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v5.7 file=build/windows/amd64/rocm_v5.7/bin/llama.dll.gz time=2024-04-20T22:04:24.150+02:00 level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v5.7 file=build/windows/amd64/rocm_v5.7/bin/ollama_llama_server.exe.gz time=2024-04-20T22:04:26.645+02:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\name\AppData\Local\Temp\ollama522826981\runners\cpu time=2024-04-20T22:04:26.645+02:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\name\AppData\Local\Temp\ollama522826981\runners\cpu_avx time=2024-04-20T22:04:26.645+02:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\name\AppData\Local\Temp\ollama522826981\runners\cpu_avx2 time=2024-04-20T22:04:26.645+02:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\name\AppData\Local\Temp\ollama522826981\runners\cuda_v11.3 time=2024-04-20T22:04:26.645+02:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=C:\Users\name\AppData\Local\Temp\ollama522826981\runners\rocm_v5.7 time=2024-04-20T22:04:26.645+02:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [rocm_v5.7 cpu cpu_avx cpu_avx2 cuda_v11.3]" time=2024-04-20T22:04:26.645+02:00 level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"

@dhiltgen
Copy link
Collaborator

@liquorLiu interesting. So our bundled v11 cuda library didn't work on your system, but removing it so it used the v12(?) version you already had installed fixed the problem, and it is now running on GPU. Is that correct? If so, I'm glad we now have a workaround, but I want to understand why this isn't working, since nvidia claims to support backwards compatibility so v11 libraries should work on newer drivers.

@alecvern alecvern mentioned this issue May 2, 2024
Closed
@dhiltgen
Copy link
Collaborator

dhiltgen commented May 4, 2024

Can folks hitting this issue try upgrading to 0.1.33? It looks like at least some of you are likely going to hit #4008 which can be worked around by removing the PhysX directory from your PATH.

@alecvern
Copy link

alecvern commented May 5, 2024 •
edited

Can folks hitting this issue try upgrading to 0.1.33? It looks like at least some of you are likely going to hit #4008 which can be worked around by removing the PhysX directory from your PATH.

Hello, @dhiltgen!
I can confirm that after 0.1.33 problem with PhysX is still presented (and in some way become even worse). The thing is: after 0.1.33 ollama tends to prefer more recent Nvidia libs, which leads to detecting cudart.dll from PhysX folder (PhysX installing alongside drivers and almost always will be selected as preferable in comparison to libs included to portable ollama release from "ollama-windows-amd64.zip").
Screenshot 2024-05-05 210229

Please, take a look at PR #4106. You and I almost simultaneously created 2 PRs solving the same problem, but I also added one more important check to the code (presence of three linked Nvidia libraries: cudart, cublas and cubasLt).
Also, in the second commit to PR I adapted your version of anti-PhysX solution into my PR (original version was not drive letter agnostic). I hope you don't mind!

@dhiltgen
Copy link
Collaborator

dhiltgen commented May 5, 2024

Thanks @alecvern !

We're not sure yet what the right approach is - intentionally skip PhysX, or pivot to the Driver API for discovery of the Nvidia GPUs. We've reached out to contacts at Nvidia and we're aiming to get closure on this before we cut the next release so we can get past this PhysX problem.

@dhiltgen
Copy link
Collaborator

There were a few different topics on this issue, but I believe we've resolved the underlying problems in recent releases. If anyone is still having problems, please make sure to update to the latest version, and if that doesn't resolve it, please share your server log and I'll re-open the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dhiltgen dhiltgen

Labels
bug Something isn't working windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@dhiltgen @alecvern @ethanlchristensen @NAME0x0 @liquorLiu @timfduffy