How to stop Qwen 3.8 Flash Next from consuming ALL the system ram?

[mkultra333]

I'm on windows 11 running the latest llama.cpp [b10850]. I have 16GB vram and 64GB system ram. I can run Qwen 3.8 Flash Next, Qwen3.8-Flash-Next-UD-Q4_K_XL.gguf, but I can't find any way to limit how much system ram it uses. It always saturates the entire system ram, leaving nothing for any additional apps I might want to run (for instance, if I'm using qwen to code an app and need to run it while we work on it.)

I've asked Claude and others to help but they can't work it out. I'd want to keep at least 8-16 GB of system ram free for other things. We've tried a bunch of settings, my latest launch .bat file is below, but there have been a lot of experiments involving mmap and stuff. Nothing has worked.

@echo off
setlocal

:: Set the directory to your Llama.cpp folder
cd /d "G:\LlamaCPPServer"

set MODEL_PATH=D:\LLM_Small\Qwen3.8-Flash-Next-UD-Q4_K_XL.gguf
set CTX_SIZE=8192
set THREADS=4
set GPU_LAYERS=99
set CPU_MOE_LAYERS=48
set SLOTS=1
set ALIAS=qwen3.8-flash-next-restricted
set PORT=8080

echo Starting Qwen3.8-Flash-Next (SSD/RAM Streaming Mode)
echo Threads:       %THREADS%
echo Context:       %CTX_SIZE%
echo GPU Layers:    %GPU_LAYERS% (non-expert weights)
echo CPU MoE:       %CPU_MOE_LAYERS% layers' experts kept off VRAM
echo.
echo Chat UI will be available at: http://localhost:8080
echo.

llama-server.exe ^
 --alias "%ALIAS%" ^
 -m "%MODEL_PATH%" ^
 -ngl %GPU_LAYERS% ^
 --n-cpu-moe %CPU_MOE_LAYERS% ^
 -ot per_layer_token_embd=CPU ^
 --no-repack ^
 -c %CTX_SIZE% ^
 --kv-unified ^
 --cache-type-k q4_0 ^
 --cache-type-v q4_0 ^
 --batch-size 256 ^
 --ubatch-size 256 ^
 -fa on ^
 -np %SLOTS% ^
 -t %THREADS% ^
 --jinja ^
 --temp 0.7 ^
 --top-p 0.80 ^
 --top-k 20 ^
 --min-p 0.0 ^
 --presence-penalty 1.5 ^
 --repeat-penalty 1.0 ^
 --port %PORT%

pause

[Recursive-Logic-Core]

Quick hint to take back to Claude — no follow-up support provided:

Your RAM saturation is caused by memory duplication and OS pagefile thrashing between --no-repack, unpinned MoE offloading, and Windows file caching on GGUF mmap.

Have Claude refactor your launch arguments with these constraints:

1. Strip --no-repack and --ot per_layer_token_embd=CPU (they fragment host buffers across your 48 CPU MoE layers).
2. Add --no-mmap to force llama-server to load strictly into allocated RAM instead of letting Windows aggressively commit your entire 64GB cache pool to virtual memory.
3. Pin your worker threads with --threads 4 --threads-batch 4 to leave host headroom for your dev tools.

[Volunteer-1]

Get more RAM. You need at least 128GB.

[mkultra333]

Both the above answers were helpful. I needed to use a smaller quant, and I needed to use the new settings.
New settings on their own didn't help, the Q4_K_XL seems to be just too big, one of the AIs suggested it's to do with the size of the lookup table. So I went with a much smaller quant IQ2_M plus new settings, and this seemed to fix the issue. I did some tests, I wouldn't trust they are consistent from run to run, since one example shows different results for two different runs.

Context   n-cpu-moe   vram  sysram  t/s
No LLM    -           0.5*  10.0    -
65K 1st   40          5.3   56.0    11.2
65K 2nd   40          12.1  48.6    14.2
65K       36          15.5  45.3    15.2
65K       32          14.4  46.5    5.7
65K       24          14.1  45.5    5.2
128K      36          15.5  45.9    8.3
*baseline free vram varied from run to run, ranging from 0.3 to 1.0. That might be why the first two runs had different results.

n_cpu_moe 36 seems the best option, fastest tokens per second and still leaves me plenty of system ram.
The model I used was Qwen3.8-Flash-Next-Uncensored-IQ2_M-00001-of-00002.gguf (2 parts, about 80GB total). Below is my batch file, in case it's of help to anyone else or someone can see a way to speed up the tokens per second.

@echo off
setlocal

:: Set the directory to your Llama.cpp folder
cd /d "G:\LlamaCPPServer"

set MODEL_PATH=D:\LLM_Small\Qwen3.8-Flash-Next-Uncensored-IQ2_M-00001-of-00002.gguf
set CTX_SIZE=65536
set GPU_LAYERS=99
set CPU_MOE_LAYERS=36
set THREADS=4
set SLOTS=1
set ALIAS=qwen3.8-flash-next
set PORT=8080

echo Starting Qwen3.8-Flash-Next-Uncensored-IQ2_M-00001-of-00002.gguf (SSD/RAM Streaming Mode)
echo Threads:       %THREADS%
echo Context:       %CTX_SIZE%
echo GPU Layers:    %GPU_LAYERS% (non-expert weights)
echo CPU MoE:       %CPU_MOE_LAYERS% layers' experts kept off VRAM
echo.
echo Chat UI will be available at: http://localhost:8080
echo.

llama-server.exe ^
 --alias "%ALIAS%" ^
 -m "%MODEL_PATH%" ^
 -ngl %GPU_LAYERS% ^
 --n-cpu-moe %CPU_MOE_LAYERS% ^
 -c %CTX_SIZE% ^
 --no-mmap ^
 --kv-unified ^
 --cache-type-k q4_0 ^
 --cache-type-v q4_0 ^
 --batch-size 256 ^
 --ubatch-size 256 ^
 -fa on ^
 -np %SLOTS% ^
 --threads %THREADS% ^
 --threads-batch %THREADS% ^
 --jinja ^
 --temp 0.7 ^
 --top-p 0.80 ^
 --top-k 20 ^
 --min-p 0.0 ^
 --presence-penalty 1.5 ^
 --repeat-penalty 1.0 ^
 --port %PORT%

pause