Ollama not using NVIDIA GPUs with gpt-oss models

[nadamas2000]

What is the issue?

Hello,

I've noticed an issue with GPU utilization. When running the gpt-oss:20b and gpt-oss:120b models, Ollama seems to be running them entirely on the CPU.

My NVIDIA GPUs (RTX 4070-Ti 16GB and RTX 3060 12GB) remain completely idle according to nvidia-smi and Task Manager, while my CPU usage is maxed out. I would expect these models to be loaded onto the GPUs for accelerated performance.

Key Information:

• Ollama Version: 0.11.0
• Operating System: Windows 11
• Intel(R) Core(TM) Ultra 7 265K (3.90 GHz) 20 cores
• Nvidia driver: Game Ready 580.88
• Models affected: gpt-oss:20b, gpt-oss:120b

Steps to Reproduce:

• ollama run gpt-oss:120b "Write a long story"
• Observe that GPU utilization is at 0% and CPU is at 100%.

Thanks for your great work on this project. Let me know if you need any more information.

server.log

[russellmm]

server.log
Can confirm. Same issue for me. Swapped over to qwen3:30b just to be sure and it is using the GPU fine.

[Shawneau]

Can confirm not working in Docker on Nvidia gpu, while other models load fine. Host is Ubuntu 22.something

[jessegross]

Can you please post the server logs?

[av]

@jessegross , sorry for the extra pull logs in the middle

Details

harbor.ollama  | time=2025-08-05T18:39:37.230Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
harbor.ollama  | time=2025-08-05T18:39:37.380Z level=INFO source=types.go:130 msg="inference compute" id=GPU-4f549573-5491-abe4-bcf8-8804171f6b2b library=cuda variant=v12 compute=8.9 driver=12.9 name="NVIDIA GeForce RTX 4090 Laptop GPU" total="15.6 GiB" available="15.3 GiB"
harbor.ollama  | [GIN] 2025/08/05 - 18:39:38 | 200 |      80.352µs |      172.22.0.3 | HEAD     "/"
harbor.ollama  | [GIN] 2025/08/05 - 18:39:39 | 200 |  650.019494ms |      172.22.0.3 | POST     "/api/pull"
harbor.ollama  | [GIN] 2025/08/05 - 18:39:40 | 200 |      25.962µs |      172.22.0.4 | HEAD     "/"
harbor.ollama  | time=2025-08-05T18:39:40.835Z level=INFO source=download.go:177 msg="downloading b112e727c6f1 in 16 861 MB part(s)"
harbor.ollama  | time=2025-08-05T18:41:21.300Z level=INFO source=download.go:295 msg="b112e727c6f1 part 13 attempt 0 failed: unexpected EOF, retrying in 1s"
harbor.ollama  | time=2025-08-05T18:41:50.892Z level=INFO source=download.go:295 msg="b112e727c6f1 part 6 attempt 0 failed: unexpected EOF, retrying in 1s"
harbor.ollama  | [GIN] 2025/08/05 - 18:43:29 | 200 |      14.745µs |      172.22.0.5 | HEAD     "/"
harbor.ollama  | [GIN] 2025/08/05 - 18:43:35 | 200 |  6.075061495s |      172.22.0.5 | POST     "/api/pull"
harbor.ollama  | time=2025-08-05T18:44:12.210Z level=INFO source=download.go:177 msg="downloading 51468a0fd901 in 1 7.4 KB part(s)"
harbor.ollama  | time=2025-08-05T18:44:13.589Z level=INFO source=download.go:177 msg="downloading d8ba2f9a17b3 in 1 18 B part(s)"
harbor.ollama  | time=2025-08-05T18:44:14.979Z level=INFO source=download.go:177 msg="downloading fcaef9305bb6 in 1 415 B part(s)"
harbor.ollama  | [GIN] 2025/08/05 - 18:44:22 | 200 |         4m42s |      172.22.0.4 | POST     "/api/pull"
harbor.ollama  | [GIN] 2025/08/05 - 18:44:28 | 200 |    27.91755ms |      172.22.0.3 | GET      "/api/tags"
harbor.ollama  | [GIN] 2025/08/05 - 18:44:28 | 200 |      84.839µs |      172.22.0.3 | GET      "/api/ps"
harbor.ollama  | [GIN] 2025/08/05 - 18:44:28 | 200 |      35.083µs |      172.22.0.3 | GET      "/api/version"
harbor.ollama  | time=2025-08-05T18:44:31.245Z level=INFO source=server.go:135 msg="system memory" total="62.4 GiB" free="52.0 GiB" free_swap="20.0 GiB"
harbor.ollama  | time=2025-08-05T18:44:31.245Z level=INFO source=server.go:175 msg=offload library=cuda layers.requested=-1 layers.model=25 layers.offload=0 layers.split="" memory.available="[15.3 GiB]" memory.gpu_overhead="0 B" memory.required.full="12.9 GiB" memory.required.partial="0 B" memory.required.kv="1.1 GiB" memory.required.allocations="[0 B]" memory.weights.total="11.7 GiB" memory.weights.repeating="10.7 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="8.0 GiB" memory.graph.partial="16.0 GiB"
harbor.ollama  | time=2025-08-05T18:44:31.273Z level=INFO source=server.go:438 msg="starting llama server" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-b112e727c6f18875636c56a779790a590d705aec9e1c0eb5a97d51fc2a778583 --ctx-size 32768 --batch-size 512 --threads 8 --no-mmap --parallel 4 --port 35337"
harbor.ollama  | time=2025-08-05T18:44:31.273Z level=INFO source=sched.go:481 msg="loaded runners" count=1
harbor.ollama  | time=2025-08-05T18:44:31.274Z level=INFO source=server.go:598 msg="waiting for llama runner to start responding"
harbor.ollama  | time=2025-08-05T18:44:31.274Z level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding"
harbor.ollama  | time=2025-08-05T18:44:31.282Z level=INFO source=runner.go:925 msg="starting ollama engine"
harbor.ollama  | time=2025-08-05T18:44:31.282Z level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:35337"
harbor.ollama  | time=2025-08-05T18:44:31.314Z level=INFO source=ggml.go:92 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=315 num_key_values=30
harbor.ollama  | ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
harbor.ollama  | ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
harbor.ollama  | ggml_cuda_init: found 1 CUDA devices:
harbor.ollama  |   Device 0: NVIDIA GeForce RTX 4090 Laptop GPU, compute capability 8.9, VMM: yes
harbor.ollama  | load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so
harbor.ollama  | load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-alderlake.so
harbor.ollama  | time=2025-08-05T18:44:31.361Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX_VNNI=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
harbor.ollama  | time=2025-08-05T18:44:31.422Z level=INFO source=ggml.go:367 msg="offloading 0 repeating layers to GPU"
harbor.ollama  | time=2025-08-05T18:44:31.422Z level=INFO source=ggml.go:371 msg="offloading output layer to CPU"
harbor.ollama  | time=2025-08-05T18:44:31.422Z level=INFO source=ggml.go:378 msg="offloaded 0/25 layers to GPU"
harbor.ollama  | time=2025-08-05T18:44:31.422Z level=INFO source=ggml.go:381 msg="model weights" buffer=CPU size="12.8 GiB"
harbor.ollama  | time=2025-08-05T18:44:31.525Z level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model"
harbor.ollama  | time=2025-08-05T18:44:31.698Z level=INFO source=ggml.go:672 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 size="0 B"
harbor.ollama  | time=2025-08-05T18:44:31.698Z level=INFO source=ggml.go:672 msg="compute graph" backend=CPU buffer_type=CPU size="8.0 GiB"
harbor.ollama  | time=2025-08-05T18:44:32.787Z level=INFO source=server.go:637 msg="llama runner started in 1.51 seconds"

Similar setup, Ollama v0.11 + Docker, other models use GPU as expected

[hrz6976]

Same here on 4xL40s.

Details

time=2025-08-05T18:48:36.321Z level=INFO source=sched.go:546 msg="updated VRAM based on existing loaded models" gpu=GPU-464c9e2c-5e57-838c-e947-f75970e572bd library=cuda total="44.4 GiB" available="43.6 GiB"
time=2025-08-05T18:48:36.321Z level=INFO source=sched.go:546 msg="updated VRAM based on existing loaded models" gpu=GPU-b41cc73c-5bc1-2795-95ad-ec87002c38e2 library=cuda total="44.4 GiB" available="43.6 GiB"
time=2025-08-05T18:48:36.321Z level=INFO source=sched.go:546 msg="updated VRAM based on existing loaded models" gpu=GPU-b7ad3d9e-25dc-58ee-8b74-aaa56e955517 library=cuda total="44.4 GiB" available="43.6 GiB"
time=2025-08-05T18:48:36.321Z level=INFO source=sched.go:546 msg="updated VRAM based on existing loaded models" gpu=GPU-81971a31-64fe-a071-c6a5-de5dc026e0f7 library=cuda total="44.4 GiB" available="43.6 GiB"
time=2025-08-05T18:48:41.318Z level=INFO source=server.go:135 msg="system memory" total="755.5 GiB" free="688.2 GiB" free_swap="60.0 GiB"
time=2025-08-05T18:48:41.318Z level=INFO source=server.go:175 msg=offload library=cuda layers.requested=-1 layers.model=37 layers.offload=0 layers.split="" memory.available="[44.0 GiB 44.0 GiB 44.0 GiB 44.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="86.7 GiB" memory.required.partial="0 B" memory.required.kv="27.0 GiB" memory.required.allocations="[0 B 0 B 0 B 0 B]" memory.weights.total="59.7 GiB" memory.weights.repeating="58.6 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="384.0 GiB" memory.graph.partial="384.0 GiB"
time=2025-08-05T18:48:41.361Z level=INFO source=server.go:438 msg="starting llama server" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-90a618fe6ff21b09ca968df959104eb650658b0bef0faef785c18c2795d993e3 --ctx-size 524288 --batch-size 512 --threads 112 --no-mmap --parallel 64 --port 32909"
time=2025-08-05T18:48:41.362Z level=INFO source=sched.go:481 msg="loaded runners" count=1
time=2025-08-05T18:48:41.362Z level=INFO source=server.go:598 msg="waiting for llama runner to start responding"
time=2025-08-05T18:48:41.362Z level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server not responding"
time=2025-08-05T18:48:41.378Z level=INFO source=runner.go:925 msg="starting ollama engine"
time=2025-08-05T18:48:41.379Z level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:32909"
time=2025-08-05T18:48:41.451Z level=INFO source=ggml.go:92 msg="" architecture=gptoss file_type=MXFP4 name="" description="" num_tensors=471 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA L40S, compute capability 8.9, VMM: yes
  Device 1: NVIDIA L40S, compute capability 8.9, VMM: yes
  Device 2: NVIDIA L40S, compute capability 8.9, VMM: yes
  Device 3: NVIDIA L40S, compute capability 8.9, VMM: yes
time=2025-08-05T18:48:41.614Z level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model"
load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so
load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so
time=2025-08-05T18:48:41.849Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
time=2025-08-05T18:48:41.955Z level=INFO source=ggml.go:367 msg="offloading 0 repeating layers to GPU"
time=2025-08-05T18:48:41.955Z level=INFO source=ggml.go:371 msg="offloading output layer to CPU"
time=2025-08-05T18:48:41.955Z level=INFO source=ggml.go:378 msg="offloaded 0/37 layers to GPU"
time=2025-08-05T18:48:41.955Z level=INFO source=ggml.go:381 msg="model weights" buffer=CPU size="60.8 GiB"
time=2025-08-05T18:48:53.769Z level=INFO source=ggml.go:672 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 size="0 B"
time=2025-08-05T18:48:53.769Z level=INFO source=ggml.go:672 msg="compute graph" backend=CUDA1 buffer_type=CUDA1 size="0 B"
time=2025-08-05T18:48:53.769Z level=INFO source=ggml.go:672 msg="compute graph" backend=CUDA2 buffer_type=CUDA2 size="0 B"
time=2025-08-05T18:48:53.769Z level=INFO source=ggml.go:672 msg="compute graph" backend=CUDA3 buffer_type=CUDA3 size="0 B"
time=2025-08-05T18:48:53.769Z level=INFO source=ggml.go:672 msg="compute graph" backend=CPU buffer_type=CPU size="128.0 GiB"
time=2025-08-05T18:48:56.185Z level=INFO source=server.go:637 msg="llama runner started in 14.82 seconds"

[jessegross]

@av @hrz6976

It looks like you both increased OLLAMA_NUM_PARALLEL. I would recommend leaving it at the default setting as higher values use more VRAM and reduce ability to offload.

[nadamas2000]

Thanks for the suggestion. I've confirmed that I'm using OLLAMA_NUM_PARALLEL=1. I have updated the issue description with the latest logs.

[Shawneau]

I have a feeling everyone experiencing this is hitting Ollama via OpenWebUI? Command line Ollama works with 100% GPU, but hitting it with open web ui goes 100% CPU

[av]

Understandable!

Setting OLLAMA_NUM_PARALLEL=1 the split is now:

NAME           ID              SIZE     PROCESSOR          CONTEXT    UNTIL              
gpt-oss:20b    05afbac4bad6    18 GB    12%/88% CPU/GPU    8192       4 minutes from now    

With OLLAMA_NUM_PARALLEL=4, it looks like:

NAME           ID              SIZE     PROCESSOR    CONTEXT    UNTIL              
gpt-oss:20b    05afbac4bad6    13 GB    100% CPU     8192       4 minutes from now    

So, possibly something is off with either ps or the estimator, as clearly batching should allocate more memory.

In both instances it only uses ~12.9 GB of VRAM, leaving some space unallocated, I hope there's some way to use that and improve the performance a bit.

[jessegross]

@nadamas2000 It looks like you increased the context length, this has a similar effect to increasing NUM_PARALLEL. You'll need to use a lower value or the default.

[nadamas2000]

Ok, in my case, with 4k context GPUs running well.
Thanks.

[hrz6976]

Thanks for spotting this! I misunderstood how OLLAMA_NUM_PARALLEL works (related: #4170). It worked after removing OLLAMA_NUM_PARALLEL from envvars. 😄
P.S. Is there a way for ollama itself calculate how many requests it can handle before falling back to CPU? I can't find a optimal OLLAMA_NUM_PARALLEL as it applies to all models and I sometimes need to run different models in parallel

[HuChundong]

i have 4x2080ti 22gb, all 88GB, gpt-oss 120b use 10% cpu, ctx is 8k, 88GB is not enough for 120B model?

[russellmm]

It was context size for me. On a 5090, I can set the context windows to 32K and the model uses the GPU. Setting to 64K and it switches to the CPU.