Qwen3.8-Flash-Next-UD-IQ4_XS started to get out of memory in unsloth desktop

[konst-sh]

I have downloaded this unsloth ggufs right after the release - before the mtp files were published and successfully run it in unsloth desktop, then the mtp arrived and I have just downloaded the mtp-Qwen3.8-Flash-Next-shared-Q8_0, put in ./snapshots//MTP/ and was able to run the model again just fine.
Now after few weeks, unsloth app just told me I have a partial model files, and it started the downloading, created the folder with new hash, reloaded the whole ~100gb of files with no option to pick exact mtp version, so it has now mtp-Qwen3.8-Flash-Next-Q8_0.gguf - which is larger in size - and now It just either breaks on model loading or on promt submissin with Unsloth app out of memory errors.
I have tried to swap the -shared MTP back instead of the one it forcefully loaded, but the doesn't see it, even thiough I have used the sha hash name and shortcut, trying to mimic the unsloth's naming and location convention - it just says I have a partial files again, and starts loading the 4.1 gb mtp file. What I'm missing here?
I'm running latest unsloth desktop app v0.1.814-beta on ubuntu 26.04 on strix halo 395+ with 128gb with 96gb vram with dynamic memory alocation option in BIOS. I was clearly able to run this model with mtp just few weeks ago.

[159753a52]

Some memory arithmetic that may explain why the bigger MTP file tips it over. This assumes the 96 GB is a fixed BIOS carve-out, so the OS and everything CPU-side live in the remaining ~32 GB.

From the tensor sizes of UD-IQ4_XS (87.2 GiB across the 3 shards):

• 26.8 GiB is the n-gram table (per_layer_token_embd.weight) and 0.63 GiB the token embedding. llama.cpp keeps both on the CPU side, so they count against the 32 GB, not the 96.
• 55.4 GiB of routed experts + ~4.4 GiB of other weights go to the GPU: ~59.8 GiB.
• KV is small for this model: only 12 of the 48 layers keep a KV cache, 12 x 2 KV heads x 256 head_dim x 2 (K+V) x 2 bytes = 24 KiB/token, so 128K ctx = 3 GiB and 256K = 6 GiB at f16.

So the GPU side is ~60 GiB + the MTP model (~3.8 GiB for a 4.1 GB file) + KV + compute buffers, roughly 70 GiB even at 256K. That fits in 96 with room to spare. The CPU side is the tight one: 27.4 GiB for the table and embedding out of ~32 GB leaves ~4 GB for the OS, the desktop app and its backend. If the table ends up resident instead of being paged from disk, the extra GB or so from the non-shared MTP (presumably it carries its own copies of tensors the -shared one reuses from the main model) is enough to OOM.

Two things worth checking (estimates, I haven't run this on a Strix Halo):

• The llama-server log under ~/.unsloth/studio/logs/llama-server/ should have a line like tensor per_layer_token_embd.weight (size = ... MiB) lazy read enabled. Upstream --lazy-mode defaults to auto (on-demand reads for tensors over 4 GiB, needs mmap). If that line is missing, the whole 26.8 GiB is being held in RAM.
• Dropping the carve-out to ~80 GB would still leave ~10 GiB of headroom on the GPU side at 256K ctx, and gives the CPU side ~48 GB, which should be enough even if the table stays resident.

[konst-sh]

@159753a52 thanks, I have somehow partially resolved this by switching to 512mb bios vram carve-out and gtt specifcation in grub. Though I feel that i'm still having some memory over-consumption compared to earlier unsloth versions where it was just running up to almost 200k context I'm now having OOM at ~170k + 16gb swap is 100% busy.

[159753a52]

Good to hear the GTT route got it loading again. With a 512 MB carve-out everything comes out of one ~125 GiB pool, so here's what the model itself should take (estimates from the file sizes and config, not measured on a Strix Halo):

• UD-IQ4_XS, all three shards: 87.25 GiB. CPU-side and GPU-side tensors now come out of the same RAM.
• MTP head: 3.85 GiB for the self-contained mtp-Qwen3.8-Flash-Next-Q8_0.gguf, vs 2.60 GiB for -shared-Q8_0.
• KV at f16: 12 full-attention layers x 2 KV heads x 256 x (K+V) x 2 bytes = 24 KiB/token, so 170K ~ 3.9 GiB and 200K ~ 4.6 GiB. The MTP layer adds ~2 KiB/token.
• Compute buffers on top, usually 1-2 GiB.

That's ~96-97 GiB at 170K, which should leave a lot more room than you're seeing. Whatever is using the rest, the shift from ~200K to ~170K lines up with the head: going from 200K to 170K only frees ~0.7 GiB, while the head Studio re-downloaded is 1,350,860,864 bytes (1.26 GiB) bigger than the shared one. That's worth ~55K tokens of f16 KV. If Studio lets you set the KV cache type, q8_0 brings 170K down to ~2.1 GiB and wins back more than the head cost.

The 16 GB of swap filling up is the part the math doesn't explain. ~97 GiB shouldn't push a 128 GB machine into swap unless something is held twice. Two things in the newest log under ~/.unsloth/studio/logs/llama-server/ would show it:

• the model buffer size lines: the big CPU-side part should be CPU_Mapped (file-backed, so the kernel can drop it instead of swapping it). If it's plain CPU, the ~27 GiB n-gram table is sitting in anonymous memory, and that's what ends up in swap.
• the per_layer_token_embd.weight ... lazy read enabled line.

cat /sys/class/drm/card*/device/mem_info_gtt_used during a long prompt shows how much the GPU side actually took.

[konst-sh]

@159753a52
Well, I switched the log to verbosity level 4, and this is what I'm getting while reruning the chat with overall size of ~170k tokens
0.04.174.327 I load_tensors: CPU model buffer size = 27465.95 MiB
0.04.174.328 I load_tensors: Vulkan0 model buffer size = 61222.07 MiB
0.04.174.329 I load_tensors: Vulkan_Host model buffer size = 644.14 MiB

/$ cat /sys/class/drm/card*/device/mem_info_gtt_used
86354481152

per_layer_token_embd.weight - there is no such line in the log at that verbosity level

btw this is what free-h shows me currently:
/$ free -h
total used free shared buff/cache available
Mem: 122Gi 106Gi 11Gi 185Mi 5.6Gi 16Gi
Swap: 15Gi 15Gi 66Mi

and these are my gtt grub options: amd_iommu=pt ttm.pages_limit=29360128 ttm.page_pool_size=15728640 - its not too agressive, but from my undestanding the model should still fit without squeezing whole 16 gb to swap

[159753a52]

Thanks, those lines settle most of it, and they also show my estimate last time was ~10 GiB short.

The weights match the file to the MiB: Vulkan0 61222.07 MiB is the non-expert tensors plus all routed experts of UD-IQ4_XS, CPU 27465.95 MiB is the n-gram table (28,800,138,240 bytes), and Vulkan_Host 644.14 MiB is the token embedding.

The name of the CPU buffer is the key: it's CPU, not CPU_Mapped. The 26.8 GiB table has been copied into anonymous RAM, which the kernel can only swap out, not drop. Add the 80.4 GiB of GTT you measured and llama-server alone holds ~107 GiB of 122, which is why the rest of the system ends up in swap.

I think I found why mmap is off. llama.cpp now defaults to --load-mode auto, which disables mmap when a device says it can't use it, and the Vulkan backend says that for every integrated GPU (ggml-org/llama.cpp#26081). Lazy reads need mmap too, which is why there's no lazy read enabled line. The loading model tensors line near the top of the log should end in (load_mode = none) if that's what happened.

What I'd try (untested on a Strix Halo):

1. Add --load-mode mmap under Advanced settings > Extra Arguments and reload. The table should then load as CPU_Mapped and be read on demand. Upstream turned mmap off on iGPUs because loading then briefly holds the GPU weights twice, so expect a slower load.
2. Set Cache RAM to 0. llama.cpp's default is an 8 GiB host prompt cache, and a cached 170K prompt for this model is ~4 GiB, so two long chats fill it. The cost is re-processing a chat when you switch back to it.
3. KV Cache Dtype q8_0 (~1.9 GiB less at 170K) and the -shared MTP head (1.26 GiB smaller).

On the gap: GTT is ~20 GiB above the GPU weights, where I expected ~9-10 (MTP head 3.85 + ~4.2 KV at 170K + compute buffers). The llama_kv_cache, llama_memory_recurrent and compute buffer size lines, plus the MTP model's model buffer size lines, would show where the rest goes.

[konst-sh]

@159753a52 thanks I should note, yesterday in conversation with that qwen 3.8 itself it proposed me to change the load mode to mmap as well, and it seem to resolve memory consumption: -lm mmap --lazy-mode off - with this I have numbers as following at context of 200k

/$ free -h
total used free shared buff/cache available
Mem: 122Gi 90Gi 9.8Gi 121Mi 23Gi 32Gi
Swap: 15Gi 3.3Gi 12Gi

Not sure, though if it will impose any performance downsides. Thanks for your help!

[Viavcious-cloud]

The non-shared Q8_0 MTP draft weights inflate base VRAM allocation by an extra ~4-6 GB during KV cache initialization, leaving zero headroom when prompt submission triggers tensor memory reservation.

If you cannot downgrade the MTP file through the UI, set --gpu-memory-utilization 0.85 or offload 2-4 layers to system RAM to prevent the crash.

We ran into this exact weight-bloat OOM issue and built a quick browser calculator to check exact peak VRAM footprints before loading models: https://huggingface.co/spaces/vivacious-cloud/vivacious-terminal-simulator