Unsloth Desktop issues with using Qwen 3.8 27B with NVFP4

[JAudibert]

Hi,

I am new to using unsloth, and the thing caught my attention was the release of Unsloth Desktop and its support for using NVFP4 on a windows machine. I am running a 5090, and I have been using for local inference for Android development via LM studio for the past year. I wanted to get the benefits of using the NVFP4 build of Qwen3.8 27B, so I download Unsloth Desktop to give it a try. It seems to not work "out of the box". I have been using the desktop application to server the standard 4 bit quant of Qwen in the GGUF format just fine. Is there any extra documentation i should look at to make it so I can run the NVFP4 format?

Any direction would be helpful.

Thanks.

[zw-g]

NVFP4 doesn't run on Unsloth Desktop's default engine, which is what serves your GGUFs. It needs the optional vLLM engine. On Windows, Desktop installs it for you inside its own private WSL2 environment. The NVFP4 guide on unsloth.ai only shows standalone vLLM and SGLang commands. The Desktop side is documented in studio/INFERENCE_ENGINES.md.

1. Pick unsloth/Qwen3.8-27B-NVFP4 in Chat. Desktop should show "Qwen3.8-27B-NVFP4 needs vLLM or SGLang". Click Install and load under vLLM. The same install is in the model's run settings (Inference engine → vLLM) and under Settings → System → More engines.
2. The first install turns on WSL2. Expect one administrator prompt and possibly a Windows restart, then click Install again. It downloads several GB, and the first load can take several minutes while kernels compile.
3. Use vLLM, not SGLang, for this model. The model card says SGLang only works with it on v0.5.19, and Desktop installs SGLang 0.5.20.

If you get no dialog and the load fails with "This model is quantized with compressed-tensors, which Unsloth cannot run yet...", Desktop has decided this PC can't run the engine. It needs Windows 10 21H2 (build 19044) or newer and NVIDIA driver 580 or newer (nvidia-smi shows your version), see engine_install.py and wsl_host.py. Settings → System → More engines shows the exact reason under vLLM.

On a 32 GB card like the 5090 you need Desktop v0.1.906-beta or later. That release added the dialog above and the fix that lets vLLM start Qwen3.8 on 32 GB cards. In #8861 it then ran at about 60 tokens/s on a 5090-sized memory budget.

[JAudibert]

I am running with vLLM as the default engine. I have installed 0.30.0. I am running the Desktop app at version 0.1.915-beta, and windows is running 25H2.

Here is a dump of the info from nvidia-smi

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 620.70.03 KMD Version: 621.14 CUDA UMD Version: 13.5 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5090 On | 00000000:01:00.0 On | N/A |
| 0% 47C P0 46W / 600W | 4227MiB / 32607MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

Is there anything else i need to check/do?\

[zw-g]

Thanks, the logs show it. The engine dies while capturing CUDA graphs with RuntimeError: FlashInfer backend is not available, raised from flashinfer_xqa_batch_decode_with_kv_cache. Your setup is fine; this is a Desktop bug:

• On RTX 50-series cards (sm_120), vLLM 0.30.0 decodes with FlashInfer's XQA kernel.
• vLLM only treats FlashInfer as usable when the flashinfer-cubin package is installed or nvcc is on PATH (has_flashinfer()).
• Desktop leaves flashinfer-cubin out of its vLLM environment (engine_install.py). It does ship nvcc, as CUDA_HOME=<env>/cuda, but the engine's PATH doesn't include <env>/cuda/bin (managed_engine.py). So vLLM picks XQA and then refuses to call it.

Workaround until it's fixed: put nvcc on PATH inside Desktop's WSL distro. In PowerShell, wsl -l -v lists it as Unsloth-Engines- plus 8 characters. Using that name:

wsl -d Unsloth-Engines-XXXXXXXX -u root -- ln -sf /opt/unsloth/engines/vllm/env-fba2a48a258741228da6f7993e525e5f/cuda/bin/nvcc /usr/local/bin/nvcc
wsl -d Unsloth-Engines-XXXXXXXX -u root -- nvcc --version

The second command should print the CUDA 13.0 compiler version. Then load the model again. The first load JIT-compiles the FlashInfer kernels, so it will take a few extra minutes. To undo it later, run rm /usr/local/bin/nvcc the same way.

[JAudibert]

I did those commands and i still had the model error out, here are the logs:
unsloth-logs-20261105-140054.zip

[zw-g]

That got it past the FlashInfer error. The engine now loads the weights and fails later, when vLLM sizes its KV cache:

ValueError: No available memory for the cache blocks.

This one is a VRAM budget problem:

• The checkpoint's weights alone take about 21.8 GiB, because 303 of its modules are left unquantized.
• Desktop gives vLLM whatever is free when you load, minus a 3 GiB reserve (engine_adapters.py). With the 4227 MiB your nvidia-smi showed in use, that's about 24.7 GiB. That leaves roughly 3 GiB for vLLM's profiling peak, the CUDA context and the KV cache, and it ran out.
• Your first attempt crashed while vLLM was still profiling memory, before it sized the KV cache. So this limit was there all along; the FlashInfer error just hit first.

To give vLLM more room:

1. Close everything else that uses the GPU (browser, LM Studio, overlays, games), and check that nvidia-smi shows as little memory in use as possible before loading. Each GB you free goes straight into vLLM's budget.
2. Leave "GPU memory limit" on Automatic. It can only lower the budget.
3. Start with a small Context Length, e.g. 8192. A longer context needs a bigger KV cache.

If it still fails with the GPU otherwise idle, this checkpoint doesn't fit vLLM on a 32 GB card, and the GGUF you already run on the default engine is the practical option.

[JAudibert]

I did as you said and closed every thing (even went down to one monitor) and still didn't load the modal.

I am giving you the logs again just incase there is something interesting.
unsloth-logs-20261105-154617.zip

Thanks for helping me out, I guess I won't be using NVFP4 a for qwen and gemma anytime soon than.

[zw-g]

Freeing the VRAM did help: this time vLLM had room for its cache. The load now fails one step later, in Desktop's own retry for hybrid models like this one:

1. vLLM refused its default max_num_seqs of 256, because only 70 Mamba cache blocks fit.
2. Desktop caught that and relaunched with max_num_seqs set to 70 (managed_engine.py).
3. The relaunch measured only 63 blocks, so it failed too, and Desktop retries only once per load.

The relaunch probably had less VRAM to work with: Desktop measures free VRAM at every launch, and this measurement came right after the first engine exited.

Desktop keeps that limit of 70 until it restarts, so it's worth one more load before giving up on NVFP4:

• Don't restart Desktop, and keep the same model and load settings. The limit is stored per load configuration.
• Keep the GPU as idle as it was for this attempt.

That load starts directly at 70. If vLLM measures fewer blocks again, Desktop retries once more at the new count. If you've restarted Desktop since, the first load may fail like this one, but it still stores the limit, so load a second time.

Separately, I opened #16851 so Desktop puts its own nvcc on PATH. Once it's merged and released, you won't need the symlink.