[Bug] Latest unsloth studio using cpu inference backend instead of gpu

[xyehya]

Just updated to latest version and loaded a model.
Inference is running CPU-only, nvidia-smi shows zero vram used by model, system RAM peaking, CPU usage peaking during inference --> probably llama-cpp pre-built binaries somehow used the CPU compiled versions instead of CPU?

    ~                                                                                                                                                                                                        
❯ curl -fsSL https://unsloth.ai/install.sh | sh

  🦥 Unsloth Studio Installer
  ────────────────────────────────────────────────────

  platform       linux
  deps           all system dependencies found
  venv           creating Python 3.13 virtual environment
                 /home/yk/.unsloth/studio/unsloth_studio
  venv           using environment
                 /home/yk/.unsloth/studio/unsloth_studio
                 installing PyTorch (https://download.pytorch.org/whl/cu130)...
                 installing unsloth (this may take a few minutes)...
  setup          running unsloth studio update...

  🦥 Unsloth Studio Setup
  ────────────────────────────────────────────────────
  node           v24.14.1 | npm 11.11.0
                 bun already installed (1.3.11)
                 building frontend...
                 using bun for package install (faster)
  frontend       built
  deps           [====================] 10/10  finalizing          
  deps           installed
  transformers   5.x pre-installed
                 installing prebuilt llama.cpp...
  llama.cpp      prebuilt installed and validated
                 installed release: unslothai/llama.cpp@b8740
  ────────────────────────────────────────────────────
  Unsloth Studio Installed
  ────────────────────────────────────────────────────
  launch         unsloth studio -H 0.0.0.0 -p 8888

                 Created Unsloth Studio shortcut

  Unsloth Studio installed!
  ────────────────────────────────────────────────────

  launch         starting Unsloth Studio...
Starting Unsloth Studio on http://185.97.92.61:8888
[OK] Frontend loaded from /home/yk/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/frontend/dist
INFO:     Started server process [21567]
INFO:     Waiting for application startup.
Hardware detected: CUDA — NVIDIA GeForce RTX 5080

============================================================
DEFAULT ADMIN ACCOUNT CREATED
    username: unsloth
    password saved to: /home/yk/.unsloth/studio/auth/.bootstrap_password
    Open the Studio UI to sign in and change it.
============================================================

INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8888 (Press CTRL+C to quit)

Logs:

{"timestamp": "2026-04-10T07:43:23.853525Z", "level": "info", "event": "Pre-caching helper GGUF: unsloth/Qwen3.5-4B-GGUF/Qwen3.5-4B-UD-Q4_K_XL.gguf"}
{"timestamp": "2026-04-10T07:43:24.018583Z", "level": "info", "event": "Helper GGUF cached: 1 file(s)"}

🦥 Unsloth Studio is running
────────────────────────────────────────────────────
  On this machine -- open this in your browser:
    http://127.0.0.1:8888
    (same as http://localhost:8888)

  From another device on your network / to share:
    http://185.97.92.61:8888

  API & health:
    http://127.0.0.1:8888/api
    http://127.0.0.1:8888/api/health
────────────────────────────────────────────────────
  Tip: if you are on this computer, open http://localhost:8888/ in your browser.

{"timestamp": "2026-04-10T07:43:40.538966Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/", "status_code": 200, "process_time_ms": 1.09}
{"timestamp": "2026-04-10T07:43:40.539174Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/chat", "status_code": 200, "process_time_ms": 0.66}
{"timestamp": "2026-04-10T07:43:40.597037Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/health", "status_code": 200, "process_time_ms": 0.31}
{"timestamp": "2026-04-10T07:43:40.613591Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.63}
{"timestamp": "2026-04-10T07:43:40.618111Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.43}
{"timestamp": "2026-04-10T07:43:40.629893Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.42}
{"timestamp": "2026-04-10T07:43:40.634821Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.45}
{"timestamp": "2026-04-10T07:43:40.925245Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.45}
{"timestamp": "2026-04-10T07:43:41.046848Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.57}
{"timestamp": "2026-04-10T07:43:48.459877Z", "level": "info", "event": "request_completed", "method": "POST", "path": "/api/auth/refresh", "status_code": 401, "process_time_ms": 0.7}
{"timestamp": "2026-04-10T07:43:48.509027Z", "level": "info", "event": "request_completed", "method": "POST", "path": "/api/auth/login", "status_code": 200, "process_time_ms": 37.94}
{"timestamp": "2026-04-10T07:43:48.585853Z", "level": "info", "event": "request_completed", "method": "POST", "path": "/api/auth/change-password", "status_code": 200, "process_time_ms": 74.7}
{"timestamp": "2026-04-10T07:43:48.591362Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.55}
{"timestamp": "2026-04-10T07:43:48.595900Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.47}
{"timestamp": "2026-04-10T07:43:48.601674Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.49}
{"timestamp": "2026-04-10T07:43:48.604043Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.92}
{"timestamp": "2026-04-10T07:43:50.901410Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/studio", "status_code": 200, "process_time_ms": 0.49}
{"timestamp": "2026-04-10T07:43:50.929233Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/health", "status_code": 200, "process_time_ms": 0.2}
{"timestamp": "2026-04-10T07:43:50.938854Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.45}
{"timestamp": "2026-04-10T07:43:51.309896Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/local", "status_code": 200, "process_time_ms": 3.27}
{"timestamp": "2026-04-10T07:43:51.310630Z", "level": "info", "event": "TrainingBackend initialized (subprocess mode)"}
{"timestamp": "2026-04-10T07:43:51.324836Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/local", "status_code": 200, "process_time_ms": 1.79}
{"timestamp": "2026-04-10T07:43:53.144756Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/local", "status_code": 200, "process_time_ms": 2.18}
{"timestamp": "2026-04-10T07:43:53.167986Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/auth/status", "status_code": 200, "process_time_ms": 0.48}
{"timestamp": "2026-04-10T07:43:53.256666Z", "level": "info", "event": "InferenceOrchestrator initialized (subprocess mode)"}
{"timestamp": "2026-04-10T07:43:53.578632Z", "level": "info", "event": "Top GGUF models: ['unsloth/Qwen3.5-35B-A3B-GGUF', 'unsloth/gemma-4-26B-A4B-it-GGUF', 'unsloth/Qwen3.5-9B-GGUF', 'unsloth/gemma-4-31B-it-GGUF', 'unsloth/gemma-4-E4B-it-GGUF', 'unsloth/Qwen3.5-27B-GGUF', 'unsloth/Qwen3.5-4B-GGUF', 'unsloth/gemma-4-E2B-it-GGUF', 'unsloth/Qwen3.5-122B-A10B-GGUF', 'unsloth/Qwen3.5-2B-GGUF', 'unsloth/Qwen3.5-0.8B-GGUF', 'unsloth/LTX-2.3-GGUF', 'unsloth/Qwen3-Coder-Next-GGUF', 'unsloth/gpt-oss-20b-GGUF', 'unsloth/Nemotron-3-Nano-30B-A3B-GGUF', 'unsloth/gpt-oss-120b-GGUF', 'unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF', 'unsloth/GLM-4.7-Flash-GGUF', 'unsloth/gemma-3-27b-it-GGUF', 'unsloth/gemma-3-12b-it-GGUF', 'unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF', 'unsloth/Qwen-Image-Edit-2511-GGUF', 'unsloth/Qwen3-VL-4B-Instruct-GGUF', 'unsloth/Qwen3.5-397B-A17B-GGUF', 'unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF', 'unsloth/Qwen2.5-VL-7B-Instruct-GGUF', 'unsloth/DeepSeek-R1-Distill-Qwen-14B-GGUF', 'unsloth/gemma-3-4b-it-GGUF', 'unsloth/Llama-3.2-1B-Instruct-GGUF', 'unsloth/gemma-3-270m-it-GGUF', 'unsloth/MiniMax-M2.5-GGUF', 'unsloth/Qwen3-VL-8B-Instruct-GGUF', 'unsloth/Qwen3-4B-Instruct-2507-GGUF']"}
{"timestamp": "2026-04-10T07:43:53.578686Z", "level": "info", "event": "Top hub models: ['unsloth/mistral-7b-v0.3-bnb-4bit', 'unsloth/Llama-3.1-8B-Instruct', 'unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit', 'unsloth/Qwen3-0.6B', 'unsloth/Qwen3-0.6B-unsloth-bnb-4bit', 'unsloth/GLM-4.7-Flash', 'unsloth/DeepSeek-OCR-2', 'unsloth/Qwen2.5-7B-Instruct', 'unsloth/Qwen3.5-9B', 'unsloth/gpt-oss-20b-unsloth-bnb-4bit', 'unsloth/Qwen2.5-7B-Instruct-bnb-4bit', 'unsloth/Qwen3-14B-unsloth-bnb-4bit', 'unsloth/Qwen3.5-4B', 'unsloth/Mistral-Small-3.2-24B-Instruct-2506-bnb-4bit', 'unsloth/Meta-Llama-3.1-8B-Instruct', 'unsloth/Qwen3-4B-Instruct-2507-unsloth-bnb-4bit', 'unsloth/GLM-4.7-Flash-FP8-Dynamic', 'unsloth/Qwen2.5-7B', 'unsloth/Qwen3.5-2B', 'unsloth/Qwen3-1.7B-unsloth-bnb-4bit', 'unsloth/Llama-3.2-1B-Instruct', 'unsloth/gpt-oss-20b', 'unsloth/Llama-3.2-3B-Instruct', 'unsloth/Qwen3-VL-4B-Instruct', 'unsloth/Qwen2.5-3B-Instruct-unsloth-bnb-4bit', 'unsloth/Qwen3-8B-unsloth-bnb-4bit', 'unsloth/gpt-oss-120b-BF16', 'unsloth/Qwen3.5-0.8B', 'unsloth/Qwen2-7B', 'unsloth/Llama-3.2-1B-Instruct-unsloth-bnb-4bit', 'unsloth/Qwen2.5-7B-Instruct-unsloth-bnb-4bit', 'unsloth/Qwen2.5-0.5B-unsloth-bnb-4bit', 'unsloth/gpt-oss-20b-BF16', 'unsloth/Llama-3.2-3B-Instruct-unsloth-bnb-4bit', 'unsloth/Llama-3.2-1B', 'unsloth/Qwen3-4B-bnb-4bit', 'unsloth/Qwen3-VL-2B-Instruct-unsloth-bnb-4bit', 'unsloth/llama-3-8b-bnb-4bit', 'unsloth/Qwen3-1.7B', 'unsloth/gemma-4-31B-it-unsloth-bnb-4bit']"}
{"timestamp": "2026-04-10T07:43:53.581601Z", "level": "info", "event": "Found 0 trained LoRA adapters in /home/yk/.unsloth/studio/outputs"}
{"timestamp": "2026-04-10T07:43:53.581665Z", "level": "info", "event": "Found 0 exported models in /home/yk/.unsloth/studio/exports"}
{"timestamp": "2026-04-10T07:43:53.583700Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/list", "status_code": 200, "process_time_ms": 328.25}
{"timestamp": "2026-04-10T07:43:53.583978Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/inference/status", "status_code": 200, "process_time_ms": 3.88}
{"timestamp": "2026-04-10T07:43:53.584059Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/loras", "status_code": 200, "process_time_ms": 3.92}
{"timestamp": "2026-04-10T07:43:53.584125Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/local", "status_code": 200, "process_time_ms": 3.96}
{"timestamp": "2026-04-10T07:43:54.661883Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/local", "status_code": 200, "process_time_ms": 10.81}
{"timestamp": "2026-04-10T07:43:54.662076Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/scan-folders", "status_code": 200, "process_time_ms": 9.29}
{"timestamp": "2026-04-10T07:43:54.662148Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/cached-gguf", "status_code": 200, "process_time_ms": 9.34}
{"timestamp": "2026-04-10T07:43:54.662209Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/cached-models", "status_code": 200, "process_time_ms": 9.38}
{"timestamp": "2026-04-10T07:43:54.666575Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/local", "status_code": 200, "process_time_ms": 1.77}
{"timestamp": "2026-04-10T07:43:54.666720Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/scan-folders", "status_code": 200, "process_time_ms": 0.57}
{"timestamp": "2026-04-10T07:43:58.614722Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/gguf-variants", "status_code": 200, "process_time_ms": 1.23}
{"timestamp": "2026-04-10T07:43:59.791266Z", "level": "info", "event": "Detected local GGUF model: /mnt/Data/lmstudio/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-IQ4_NL.gguf"}
{"timestamp": "2026-04-10T07:43:59.791431Z", "level": "info", "event": "request_completed", "method": "POST", "path": "/api/inference/validate", "status_code": 200, "process_time_ms": 0.91}
{"timestamp": "2026-04-10T07:43:59.799871Z", "level": "info", "event": "Detected local GGUF model: /mnt/Data/lmstudio/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-IQ4_NL.gguf"}
{"timestamp": "2026-04-10T07:43:59.855784Z", "level": "info", "event": "GGUF metadata: context_length=262144"}
{"timestamp": "2026-04-10T07:43:59.855833Z", "level": "info", "event": "GGUF metadata: chat_template=12045 chars"}
{"timestamp": "2026-04-10T07:43:59.855847Z", "level": "info", "event": "GGUF metadata: model supports reasoning (enable_thinking)"}
{"timestamp": "2026-04-10T07:43:59.855868Z", "level": "info", "event": "GGUF metadata: model supports tool calling"}
{"timestamp": "2026-04-10T07:43:59.871817Z", "level": "info", "event": "GGUF size: 12.5 GB, est. KV cache: 56.7 GB, context: 262144, GPUs free: [(0, 15838)], selected: None, fit: True"}
{"timestamp": "2026-04-10T07:43:59.871870Z", "level": "info", "event": "Reasoning model: enable_thinking=True by default"}
{"timestamp": "2026-04-10T07:43:59.871890Z", "level": "info", "event": "Starting llama-server: /home/yk/.unsloth/llama.cpp/llama-server -m /mnt/Data/lmstudio/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-IQ4_NL.gguf --port 53141 -c 262144 --parallel 1 --flash-attn on --fit on --jinja --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 48 --draft-max 64 --chat-template-kwargs {\"enable_thinking\": true}"}
{"timestamp": "2026-04-10T07:44:03.909391Z", "level": "info", "event": "llama-server ready on port 53141 for model '/home/yk/Data/lmstudio/models/unsloth/gemma-4-26B-A4B-it-GGUF'"}
{"timestamp": "2026-04-10T07:44:03.909539Z", "level": "info", "event": "Loaded GGUF model via llama-server: /home/yk/Data/lmstudio/models/unsloth/gemma-4-26B-A4B-it-GGUF"}
{"timestamp": "2026-04-10T07:44:03.915955Z", "level": "info", "event": "Loaded default model defaults from /home/yk/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/assets/configs/model_defaults/default.yaml"}
{"timestamp": "2026-04-10T07:44:03.927654Z", "level": "info", "event": "request_completed", "method": "POST", "path": "/api/inference/load", "status_code": 200, "process_time_ms": 4129.39}
{"timestamp": "2026-04-10T07:44:03.932462Z", "level": "info", "event": "Loaded default model defaults from /home/yk/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/assets/configs/model_defaults/default.yaml"}
{"timestamp": "2026-04-10T07:44:03.933551Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/list", "status_code": 200, "process_time_ms": 3.63}
{"timestamp": "2026-04-10T07:44:03.933921Z", "level": "info", "event": "Found 0 trained LoRA adapters in /home/yk/.unsloth/studio/outputs"}
{"timestamp": "2026-04-10T07:44:03.933962Z", "level": "info", "event": "Found 0 exported models in /home/yk/.unsloth/studio/exports"}
{"timestamp": "2026-04-10T07:44:03.934097Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/inference/status", "status_code": 200, "process_time_ms": 2.94}
{"timestamp": "2026-04-10T07:44:03.934229Z", "level": "info", "event": "request_completed", "method": "GET", "path": "/api/models/loras", "status_code": 200, "process_time_ms": 0.7}
{"timestamp": "2026-04-10T07:44:18.618618Z", "level": "info", "event": "request_completed", "method": "POST", "path": "/v1/chat/completions", "status_code": 200, "process_time_ms": 22.41}
^C{"timestamp": "2026-04-10T07:45:27.859173Z", "level": "info", "event": "Graceful shutdown initiated \u2014 cleaning up subprocesses..."}
{"timestamp": "2026-04-10T07:45:27.874610Z", "level": "info", "event": "All subprocesses cleaned up"}

❯ nvidia-smi
Fri Apr 10 10:48:43 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.58.03              Driver Version: 595.58.03      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5080        On  |   00000000:02:00.0 Off |                  N/A |
|  0%   33C    P8             17W /  360W |       4MiB /  16303MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

    ~                                                                                                                                                                                                        
❯ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Tue_Dec_16_07:23:41_PM_PST_2025
Cuda compilation tools, release 13.1, V13.1.115
Build cuda_13.1.r13.1/compiler.37061995_0

[rolandtannous]

@xyehya are you on windows?
if yes, the issue is obvious:

• your CUDA version shown in nvidia-smi and the cuda toolkit version should match for windows or else it won't work.
  In your case they don't . one shows 13.2 and the other 13.1
  13.2 is also not recommended. it's buggy and we reported some issues to nvidia.
  You need to downgrade your cuda version 13.1
  Use this nvidia website to do it: https://developer.nvidia.com/cuda-13-1-1-download-archive

[xyehya]

@rolandtannous I'm on linux, specifically Fedora43.
nvidia-smi reports the maximum CUDA version supported by the driver not the installed/active one.
I have toolkits for Cuda13.2 and Cuda13.1 both installed as I reported the issue related to Cuda13.2 & llama.cpp so I'm aware of the issue.
Current env variables point to Cuda13.1 so Cuda13.2 issue is not applicable
The installation script did not compile llama.cpp locally but fetched the binaries from unsloth repo --> not compilation issue
The previous studio version was working flawlessly yesterday in my current setup - no changes have been done on the system since then.
I tried both the installation script from scratch + unsloth studio update + full removal and installation -> both showing same issue.

[rolandtannous]

Not sure what's going on but i remember another community user with fedora also had issues.
wondering if a system tool we're calling using the ubuntu format (that's what we use by default) isn't fedora compatible.
do you mind sharing the exact commands you used to install? thanks

[m16a1]

Downgrading to CUDA 13.1 doesn't help. Inference is running in CPU mode.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        Off |   00000000:01:00.0 Off |                  N/A |
|  0%   33C    P8             22W /  575W |      18MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

[rolandtannous]

i was able to reproduce. looking into it

[danielhanchen]

We fixed it - llama.cpp 10 hours ago as part of ggml-org/llama.cpp#19378 forced NCCL and our pre-compiled binaries forgot to turn off NCCL.

We re-did our pre-compiled binaries with NCCL disabled.

Please delete the llama.cpp folder or use unsloth studio update which will pull in llama.cpp anew

[xyehya]

Confirmed. Working from my side too following @danielhanchen's proposition. Thanks!