Run failure on disabled GPU

[ihor-sokoliuk]

What is the issue?

Here is the BAT file to run Ollama on my Windows machine:

@echo off
set CUDA_VISIBLE_DEVICES=""
set OLLAMA_ORIGINS="*"
set OLLAMA_HOST=0.0.0.0:11434
set OLLAMA_DEBUG=1
ollama serve
timeout /t 100 /nobreak

When I run the Ollama server with the flag CUDA_VISIBLE_DEVICES="" it starts but fails in a minute or two without any error message and with a random return code like 2147842401.

Please take a look at the attached server log.

If I run the same BAT without the CUDA_VISIBLE_DEVICES variable, it works as expected.

How can I run the Ollama server without a GPU (with CPU+RAM only) so that it will continue to work?

Relevant log output

2025/02/06 11:44:15 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\ihor\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-02-06T11:44:15.892-06:00 level=INFO source=images.go:432 msg="total blobs: 7"
time=2025-02-06T11:44:15.892-06:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-02-06T11:44:15.893-06:00 level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7)"
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=common.go:80 msg="runners located" dir=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx\ollama_llama_server.exe
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v11_avx\ollama_llama_server.exe
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v12_avx\ollama_llama_server.exe
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners\rocm_avx\ollama_llama_server.exe
time=2025-02-06T11:44:15.893-06:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cuda_v12_avx rocm_avx cpu cpu_avx cpu_avx2 cuda_v11_avx]"
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=routes.go:1268 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2025-02-06T11:44:15.893-06:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
time=2025-02-06T11:44:15.893-06:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2025-02-06T11:44:15.893-06:00 level=INFO source=gpu_windows.go:183 msg="efficiency cores detected" maxEfficiencyClass=1
time=2025-02-06T11:44:15.893-06:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=24 efficiency=16 threads=32
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=gpu.go:99 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=nvml.dll
time=2025-02-06T11:44:15.893-06:00 level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvml.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvml.dll C:\\Program Files\\Python313\\Scripts\\nvml.dll C:\\Program Files\\Python313\\nvml.dll C:\\WINDOWS\\system32\\nvml.dll C:\\WINDOWS\\nvml.dll C:\\WINDOWS\\System32\\Wbem\\nvml.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\nvml.dll C:\\WINDOWS\\System32\\OpenSSH\\nvml.dll C:\\Program Files\\Microsoft VS Code\\bin\\nvml.dll C:\\Program Files (x86)\\WinSCP\\nvml.dll C:\\Program Files\\Git\\cmd\\nvml.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\nvml.dll C:\\Program Files\\Mullvad VPN\\resources\\nvml.dll C:\\Program Files\\PowerShell\\7\\nvml.dll C:\\Users\\ihor\\AppData\\Local\\Microsoft\\WindowsApps\\nvml.dll D:\\Development\\BigBoyEnhancement\\nvml.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\nvml.dll c:\\Windows\\System32\\nvml.dll]"
time=2025-02-06T11:44:15.894-06:00 level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths="[C:\\WINDOWS\\system32\\nvml.dll c:\\Windows\\System32\\nvml.dll]"
time=2025-02-06T11:44:15.903-06:00 level=DEBUG source=gpu.go:120 msg="nvidia-ml loaded" library=C:\WINDOWS\system32\nvml.dll
time=2025-02-06T11:44:15.903-06:00 level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=nvcuda.dll
time=2025-02-06T11:44:15.903-06:00 level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvcuda.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvcuda.dll C:\\Program Files\\Python313\\Scripts\\nvcuda.dll C:\\Program Files\\Python313\\nvcuda.dll C:\\WINDOWS\\system32\\nvcuda.dll C:\\WINDOWS\\nvcuda.dll C:\\WINDOWS\\System32\\Wbem\\nvcuda.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\nvcuda.dll C:\\WINDOWS\\System32\\OpenSSH\\nvcuda.dll C:\\Program Files\\Microsoft VS Code\\bin\\nvcuda.dll C:\\Program Files (x86)\\WinSCP\\nvcuda.dll C:\\Program Files\\Git\\cmd\\nvcuda.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\nvcuda.dll C:\\Program Files\\Mullvad VPN\\resources\\nvcuda.dll C:\\Program Files\\PowerShell\\7\\nvcuda.dll C:\\Users\\ihor\\AppData\\Local\\Microsoft\\WindowsApps\\nvcuda.dll D:\\Development\\BigBoyEnhancement\\nvcuda.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll c:\\windows\\system*\\nvcuda.dll]"
time=2025-02-06T11:44:15.971-06:00 level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths=[C:\WINDOWS\system32\nvcuda.dll]
initializing C:\WINDOWS\system32\nvcuda.dll
dlsym: cuInit - 00007FFF05775F80
dlsym: cuDriverGetVersion - 00007FFF05776020
dlsym: cuDeviceGetCount - 00007FFF05776816
dlsym: cuDeviceGet - 00007FFF05776810
dlsym: cuDeviceGetAttribute - 00007FFF05776170
dlsym: cuDeviceGetUuid - 00007FFF05776822
dlsym: cuDeviceGetName - 00007FFF0577681C
dlsym: cuCtxCreate_v3 - 00007FFF05776894
dlsym: cuMemGetInfo_v2 - 00007FFF05776996
dlsym: cuCtxDestroy - 00007FFF057768A6
calling cuInit
cuInit err: 100
time=2025-02-06T11:44:15.980-06:00 level=INFO source=gpu.go:620 msg="no nvidia devices detected by library C:\\WINDOWS\\system32\\nvcuda.dll"
time=2025-02-06T11:44:15.980-06:00 level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=cudart64_*.dll
time=2025-02-06T11:44:15.980-06:00 level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll C:\\Program Files\\Python313\\Scripts\\cudart64_*.dll C:\\Program Files\\Python313\\cudart64_*.dll C:\\WINDOWS\\system32\\cudart64_*.dll C:\\WINDOWS\\cudart64_*.dll C:\\WINDOWS\\System32\\Wbem\\cudart64_*.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\cudart64_*.dll C:\\WINDOWS\\System32\\OpenSSH\\cudart64_*.dll C:\\Program Files\\Microsoft VS Code\\bin\\cudart64_*.dll C:\\Program Files (x86)\\WinSCP\\cudart64_*.dll C:\\Program Files\\Git\\cmd\\cudart64_*.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\cudart64_*.dll C:\\Program Files\\Mullvad VPN\\resources\\cudart64_*.dll C:\\Program Files\\PowerShell\\7\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Microsoft\\WindowsApps\\cudart64_*.dll D:\\Development\\BigBoyEnhancement\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll c:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v*\\bin\\cudart64_*.dll]"
time=2025-02-06T11:44:15.983-06:00 level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths="[C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_110.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_12.dll]"
cudaSetDevice err: 100
time=2025-02-06T11:44:15.987-06:00 level=DEBUG source=gpu.go:592 msg="Unable to load cudart library C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_110.dll: cudart init failure: 100"
cudaSetDevice err: 100
time=2025-02-06T11:44:15.991-06:00 level=DEBUG source=gpu.go:592 msg="Unable to load cudart library C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_12.dll: cudart init failure: 100"
time=2025-02-06T11:44:15.991-06:00 level=DEBUG source=amd_windows.go:35 msg="unable to load amdhip64_6.dll, please make sure to upgrade to the latest amd driver: The specified module could not be found."
time=2025-02-06T11:44:15.991-06:00 level=INFO source=gpu.go:392 msg="no compatible GPUs were discovered"
releasing nvml library
time=2025-02-06T11:44:15.992-06:00 level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="127.7 GiB" available="117.3 GiB"

OS

Windows

GPU

Nvidia

CPU

Intel

Ollama version

0.5.7

[rick-github]

CUDA_VISIBLE_DEVICES="" is not a valid setting. Documentation says comma separated list of identifiers. You can disable all devices with CUDA_VISIBLE_DEVICES="-1".

[ihor-sokoliuk]

@rick-github Thank you for getting back to me so quickly.

I've tried that, and the same issue remains. It stops working in a minute.

The updated BAT file:

@echo off
set CUDA_VISIBLE_DEVICES="-1"
set OLLAMA_ORIGINS="*"
set OLLAMA_HOST=0.0.0.0:11434
set OLLAMA_DEBUG=1
ollama serve
timeout /t 100 /nobreak

The latest log:

2025/02/06 15:34:24 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES:-1 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\ihor\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-02-06T15:34:24.814-06:00 level=INFO source=images.go:432 msg="total blobs: 7"
time=2025-02-06T15:34:24.814-06:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-02-06T15:34:24.814-06:00 level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7)"
time=2025-02-06T15:34:24.814-06:00 level=DEBUG source=common.go:80 msg="runners located" dir=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx\ollama_llama_server.exe
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v11_avx\ollama_llama_server.exe
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v12_avx\ollama_llama_server.exe
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=common.go:124 msg="availableServers : found" file=C:\Users\ihor\AppData\Local\Programs\Ollama\lib\ollama\runners\rocm_avx\ollama_llama_server.exe
time=2025-02-06T15:34:24.815-06:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cuda_v12_avx rocm_avx cpu cpu_avx cpu_avx2 cuda_v11_avx]"
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=routes.go:1268 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2025-02-06T15:34:24.815-06:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
time=2025-02-06T15:34:24.815-06:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2025-02-06T15:34:24.815-06:00 level=INFO source=gpu_windows.go:183 msg="efficiency cores detected" maxEfficiencyClass=1
time=2025-02-06T15:34:24.815-06:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=24 efficiency=16 threads=32
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=gpu.go:99 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=nvml.dll
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvml.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvml.dll C:\\Program Files\\Python313\\Scripts\\nvml.dll C:\\Program Files\\Python313\\nvml.dll C:\\WINDOWS\\system32\\nvml.dll C:\\WINDOWS\\nvml.dll C:\\WINDOWS\\System32\\Wbem\\nvml.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\nvml.dll C:\\WINDOWS\\System32\\OpenSSH\\nvml.dll C:\\Program Files\\Microsoft VS Code\\bin\\nvml.dll C:\\Program Files (x86)\\WinSCP\\nvml.dll C:\\Program Files\\Git\\cmd\\nvml.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\nvml.dll C:\\Program Files\\Mullvad VPN\\resources\\nvml.dll C:\\Program Files\\PowerShell\\7\\nvml.dll C:\\Users\\ihor\\AppData\\Local\\Microsoft\\WindowsApps\\nvml.dll D:\\Development\\BigBoyEnhancement\\nvml.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\nvml.dll c:\\Windows\\System32\\nvml.dll]"
time=2025-02-06T15:34:24.815-06:00 level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths="[C:\\WINDOWS\\system32\\nvml.dll c:\\Windows\\System32\\nvml.dll]"
time=2025-02-06T15:34:24.824-06:00 level=DEBUG source=gpu.go:120 msg="nvidia-ml loaded" library=C:\WINDOWS\system32\nvml.dll
time=2025-02-06T15:34:24.824-06:00 level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=nvcuda.dll
time=2025-02-06T15:34:24.825-06:00 level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvcuda.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvcuda.dll C:\\Program Files\\Python313\\Scripts\\nvcuda.dll C:\\Program Files\\Python313\\nvcuda.dll C:\\WINDOWS\\system32\\nvcuda.dll C:\\WINDOWS\\nvcuda.dll C:\\WINDOWS\\System32\\Wbem\\nvcuda.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\nvcuda.dll C:\\WINDOWS\\System32\\OpenSSH\\nvcuda.dll C:\\Program Files\\Microsoft VS Code\\bin\\nvcuda.dll C:\\Program Files (x86)\\WinSCP\\nvcuda.dll C:\\Program Files\\Git\\cmd\\nvcuda.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\nvcuda.dll C:\\Program Files\\Mullvad VPN\\resources\\nvcuda.dll C:\\Program Files\\PowerShell\\7\\nvcuda.dll C:\\Users\\ihor\\AppData\\Local\\Microsoft\\WindowsApps\\nvcuda.dll D:\\Development\\BigBoyEnhancement\\nvcuda.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll c:\\windows\\system*\\nvcuda.dll]"
time=2025-02-06T15:34:24.825-06:00 level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths=[C:\WINDOWS\system32\nvcuda.dll]
initializing C:\WINDOWS\system32\nvcuda.dll
dlsym: cuInit - 00007FFC7C3D5F80
dlsym: cuDriverGetVersion - 00007FFC7C3D6020
dlsym: cuDeviceGetCount - 00007FFC7C3D6816
dlsym: cuDeviceGet - 00007FFC7C3D6810
dlsym: cuDeviceGetAttribute - 00007FFC7C3D6170
dlsym: cuDeviceGetUuid - 00007FFC7C3D6822
dlsym: cuDeviceGetName - 00007FFC7C3D681C
dlsym: cuCtxCreate_v3 - 00007FFC7C3D6894
dlsym: cuMemGetInfo_v2 - 00007FFC7C3D6996
dlsym: cuCtxDestroy - 00007FFC7C3D68A6
calling cuInit
cuInit err: 100
time=2025-02-06T15:34:24.833-06:00 level=INFO source=gpu.go:620 msg="no nvidia devices detected by library C:\\WINDOWS\\system32\\nvcuda.dll"
time=2025-02-06T15:34:24.834-06:00 level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=cudart64_*.dll
time=2025-02-06T15:34:24.834-06:00 level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll C:\\Program Files\\Python313\\Scripts\\cudart64_*.dll C:\\Program Files\\Python313\\cudart64_*.dll C:\\WINDOWS\\system32\\cudart64_*.dll C:\\WINDOWS\\cudart64_*.dll C:\\WINDOWS\\System32\\Wbem\\cudart64_*.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\cudart64_*.dll C:\\WINDOWS\\System32\\OpenSSH\\cudart64_*.dll C:\\Program Files\\Microsoft VS Code\\bin\\cudart64_*.dll C:\\Program Files (x86)\\WinSCP\\cudart64_*.dll C:\\Program Files\\Git\\cmd\\cudart64_*.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\cudart64_*.dll C:\\Program Files\\Mullvad VPN\\resources\\cudart64_*.dll C:\\Program Files\\PowerShell\\7\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Microsoft\\WindowsApps\\cudart64_*.dll D:\\Development\\BigBoyEnhancement\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll c:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v*\\bin\\cudart64_*.dll]"
time=2025-02-06T15:34:24.837-06:00 level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths="[C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_110.dll C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_12.dll]"
cudaSetDevice err: 100
time=2025-02-06T15:34:24.841-06:00 level=DEBUG source=gpu.go:592 msg="Unable to load cudart library C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_110.dll: cudart init failure: 100"
cudaSetDevice err: 100
time=2025-02-06T15:34:24.845-06:00 level=DEBUG source=gpu.go:592 msg="Unable to load cudart library C:\\Users\\ihor\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_12.dll: cudart init failure: 100"
time=2025-02-06T15:34:24.846-06:00 level=DEBUG source=amd_windows.go:35 msg="unable to load amdhip64_6.dll, please make sure to upgrade to the latest amd driver: The specified module could not be found."
time=2025-02-06T15:34:24.846-06:00 level=INFO source=gpu.go:392 msg="no compatible GPUs were discovered"
releasing nvml library
time=2025-02-06T15:34:24.846-06:00 level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="127.7 GiB" available="119.5 GiB"

A screenshot with the failure and without any message:

I have the latest Windows 11 24H2 with all possible OS and Drivers upgrades.

Any other ideas?

[rick-github]

Hmm, that works to disable CUDA devices on my linux systems, and it doesn't crash. There is another way to disable GPU, set num_gpu to zero, see here.

[ihor-sokoliuk]

The num_gpu in Modelfile approach works best for me. Thank you @rick-github!