What is the issue?
On an RTX 5090 + 5070 Ti (total 48 GB VRAM), it is running almost entirely on CPU. Only storing the KV Cache on the GPU but using just 1% of GPU while it is using all 20 cores of the CPU. That gets it about 6 tokens/second.
Diagnosed with Claude and it says "The gpt-oss:20b model runs on CPU because it uses MXFP4 quantization, which likely lacks GPU acceleration support in
Ollama. Your working models (gemma3, qwen3) use standard quantizations (Q4_0, etc.) that have full GPU support.
Solution: Look for a gpt-oss model with standard quantization (Q4_K_M, Q5_K_M, Q8_0) for GPU acceleration, or wait for
MXFP4 GPU support in future Ollama updates."
Running Ollama 0.11.3, CUDA 12.9.1, Ubuntu 24.04.2 LTS
services:
ollama:
image: ollama/ollama:0.11.3
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
environment:
- NVIDIA_VISIBLE_DEVICES=all
- OLLAMA_CONTEXT_LENGTH=32768
- OLLAMA_CONTEXT_LENGTH=49152
- OLLAMA_CONTEXT_LENGTH=65536
- OLLAMA_CONTEXT_LENGTH=131072
- OLLAMA_KV_CACHE_TYPE=q8_0
- OLLAMA_NUM_PARALLEL=1
- OLLAMA_NUM_THREADS=16
- OLLAMA_MAX_QUEUE=32
- OLLAMA_MAX_LOADED_MODELS=1
- OLLAMA_FLASH_ATTENTION=1
- OLLAMA_KEEP_ALIVE=5m
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
- ollama-data:/root/.ollama
- /mnt/raid0/medgemma27b/medgemma-27b-text-it-q4_k_m:/models/medgemma-27b-text-it-q4_k_m
volumes:
ollama-data:
Relevant log output
OS
No response
GPU
No response
CPU
No response
Ollama version
No response
What is the issue?
On an RTX 5090 + 5070 Ti (total 48 GB VRAM), it is running almost entirely on CPU. Only storing the KV Cache on the GPU but using just 1% of GPU while it is using all 20 cores of the CPU. That gets it about 6 tokens/second.
Diagnosed with Claude and it says "The gpt-oss:20b model runs on CPU because it uses MXFP4 quantization, which likely lacks GPU acceleration support in
Ollama. Your working models (gemma3, qwen3) use standard quantizations (Q4_0, etc.) that have full GPU support.
Solution: Look for a gpt-oss model with standard quantization (Q4_K_M, Q5_K_M, Q8_0) for GPU acceleration, or wait for
MXFP4 GPU support in future Ollama updates."
Running Ollama 0.11.3, CUDA 12.9.1, Ubuntu 24.04.2 LTS
services:
ollama:
image: ollama/ollama:0.11.3
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
environment:
- NVIDIA_VISIBLE_DEVICES=all
- OLLAMA_CONTEXT_LENGTH=32768
- OLLAMA_CONTEXT_LENGTH=49152
- OLLAMA_CONTEXT_LENGTH=65536
volumes:
ollama-data:
Relevant log output
OS
No response
GPU
No response
CPU
No response
Ollama version
No response