r15 just pushed that finally fixes -sm tensor when using -ncmoe > 0

[stew675]

Today I've worked hard on getting the ncmoe feature (the thing that allows you to run models larger than your available VRAM) more polished. -sm tensor, while it worked before, had issues with MTP. Those are now resolved.

Also added is fully automatic decode arena allocations. This allows people with a model like Qwen3.-8-Flash-Next to simply point your GGUF at a GPU, specify -ncmoe 48, and it will figure it out how to best make it work.

An example llama-server start command is:

HIP_VISIBLE_DEVICES=0,1 \
llama-server \
  -m  /llm/models/Qwen3.8/Flash-Next/IQ4_NL/Qwen3.8-Flash-Next-IQ4_NL-PROJFIX-00001-of-00009.gguf \
  -md /llm/models/Qwen3.8/Flash-Next/IQ4_NL/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
  -ngl 99 \
  -sm tensor \
  -ncmoe 48 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  -t 8 \
  -c 204800 -b 2048 -ub 2048 \
  -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --host 127.0.0.1 --port 8080 \
  --jinja

Substitute in the correct paths as suits your local system

There's still a few straggling items for a bit more polish, but this should get people serving up Q38FN on minimal hardware working fairly well.

[stew675]

Sigh - still finding corruption in certain scenarios.

I tracked it down to the ring buffer and the redirection itself being the issue there.

The prefill ring buffer has been broken from the outset it seems :(

[stew675]

Given all the issues, I'm hesitant to claim success. r17 is about to go out that hopefully fixes the -sm tensor issues with MTP + -ncmoe > 0 for Qwen3.8-Flash-Next. Thing is, I've said that many times before, and then something always seems to pop up. This whole campaign effort is cursed! We'll see. If that is still failing, then just drop back to -sm layer and let me know.

There's still some more items to come to round the work off.

[stew675]

I think I'm finally really close to a proper, working, tensor-split + MTP capable solution to the MoE weights on host issue.

The corruption is gone. The arena on the cards auto-sizes on the fly. Concurrent sessions works (both prefill * N, decode * N, mixed prefill/decode). Prefill with Qwen3.8-Flash-Next @ IQ4_XS on 2 x R9700's can hit up to 1900t/s with all expert weights on the host. Decode with MTP ranges anywhere from 50-90t/s depending on what you're doing. Arena hit rates get reported.

The next release is going to have all this. There will almost certainly still be some rough edges, but I'm finally seeing the light at the end of the tunnel on this absolute monster of an implementation. This upcoming release is going to be a big patch

[stew675]

r22 is now live and with the new slab allocator that fixes almost all prior issues:

I ran a server like so. Qwen3.8-Flash-Next at ~93GiB in size on a pair of R9700's each running with just 4 lanes of PCIe bandwidth to the CPU/Host. PCEie5x8 should be faster than below.

Environment: HIP_VISIBLE_DEVICES=0,1 NCCL_PROXY_CPUSET=8,9,10,11,12,13,14,15 GGML_CUDA_ALLREDUCE=ce

Starting: ~/llama.cpp/build-rocm/bin/llama-server --model /llm/models/Qwen3.8/Flash-Next/IQ4_NL/Qwen3.8-Flash-Next-IQ4_NL-PROJFIX-00001-of-00009.gguf --alias Qwen3.8-Flash-Next-IQ4_NL --fit off --top-k 20 --port 8033 --threads 7 --parallel 1 --top-p 0.95 --min-p 0.001 --verbosity 3 --host 0.0.0.0 --cpu-strict 1 --predict 98304 --cpu-range 1-7 --threads-http 4 --load-mode none --cache-ram 16384 --ctx-size 204800 --flash-attn auto --temperature 1.0 --batch-size 6144 --ubatch-size 6144 --n-gpu-layers all --no-kv-unified --cache-type-k q8_0 --cache-type-v q8_0 --ctx-checkpoints 64 --image-min-tokens 1024 --cache-idle-slots --reasoning-budget 65536 --reasoning-preserve --checkpoint-min-step 4096 --lazy-mode off --n-cpu-moe 48 --split-mode tensor --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.001 --spec-draft-model /llm/models/Qwen3.8/Flash-Next/IQ4_NL//mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --chat-template-kwargs {"reasoning_effort":"medium"}

1500t/s for prefill when loading in a ~30K token document

4.52.696.538 I slot print_timing: id  0 | task 4555 | prompt processing, n_tokens =   6144, progress = 0.22, t =   4.94 s / 1242.73 tokens per second
4.56.186.726 I slot print_timing: id  0 | task 4555 | prompt processing, n_tokens =  12288, progress = 0.42, t =   8.54 s / 1439.49 tokens per second
4.59.655.737 I slot print_timing: id  0 | task 4555 | prompt processing, n_tokens =  18432, progress = 0.63, t =  12.01 s / 1534.90 tokens per second
5.03.004.613 I slot print_timing: id  0 | task 4555 | prompt processing, n_tokens =  23452, progress = 0.79, t =  15.40 s / 1523.19 tokens per second
5.08.160.919 I slot print_timing: id  0 | task 4555 | prompt processing, n_tokens =  29592, progress = 1.00, t =  20.47 s / 1445.78 tokens per second

and 70-80t/s for decode when generating code:

7.29.964.679 I slot print_timing: id  0 | task 5533 | n_gen =   2233, tg =  61.53 t/s, tg_3s =  74.24 t/s
7.32.972.224 I slot print_timing: id  0 | task 5533 | n_gen =   2453, tg =  62.42 t/s, tg_3s =  73.15 t/s
7.35.981.174 I slot print_timing: id  0 | task 5533 | n_gen =   2677, tg =  63.27 t/s, tg_3s =  74.44 t/s
7.39.030.261 I slot print_timing: id  0 | task 5533 | n_gen =   2905, tg =  64.05 t/s, tg_3s =  74.78 t/s
7.42.043.535 I slot print_timing: id  0 | task 5533 | n_gen =   3129, tg =  64.69 t/s, tg_3s =  74.34 t/s
7.45.070.193 I slot print_timing: id  0 | task 5533 | n_gen =   3361, tg =  65.39 t/s, tg_3s =  76.65 t/s
7.48.122.956 I slot print_timing: id  0 | task 5533 | n_gen =   3597, tg =  66.06 t/s, tg_3s =  77.31 t/s
7.51.143.492 I slot print_timing: id  0 | task 5533 | n_gen =   3837, tg =  66.77 t/s, tg_3s =  79.46 t/s
7.54.175.957 I slot print_timing: id  0 | task 5533 | n_gen =   4073, tg =  67.32 t/s, tg_3s =  77.82 t/s
7.57.211.870 I slot print_timing: id  0 | task 5533 | n_gen =   4320, tg =  67.99 t/s, tg_3s =  81.36 t/s
8.00.230.214 I slot print_timing: id  0 | task 5533 | n_gen =   4568, tg =  68.63 t/s, tg_3s =  82.16 t/s
8.03.265.203 I slot print_timing: id  0 | task 5533 | n_gen =   4808, tg =  69.09 t/s, tg_3s =  79.08 t/s
8.06.287.320 I slot print_timing: id  0 | task 5533 | n_gen =   5035, tg =  69.34 t/s, tg_3s =  75.11 t/s
8.09.300.284 I slot print_timing: id  0 | task 5533 | n_gen =   5271, tg =  69.70 t/s, tg_3s =  78.33 t/s
8.12.315.871 I slot print_timing: id  0 | task 5533 | n_gen =   5507, tg =  70.03 t/s, tg_3s =  78.26 t/s
8.15.355.521 I slot print_timing: id  0 | task 5533 | n_gen =   5747, tg =  70.36 t/s, tg_3s =  78.96 t/s
8.18.403.934 I slot print_timing: id  0 | task 5533 | n_gen =   5987, tg =  70.66 t/s, tg_3s =  78.73 t/s
8.21.455.575 I slot print_timing: id  0 | task 5533 | n_gen =   6215, tg =  70.80 t/s, tg_3s =  74.71 t/s
8.24.473.269 I slot print_timing: id  0 | task 5533 | n_gen =   6438, tg =  70.90 t/s, tg_3s =  73.90 t/s
8.27.483.854 I slot print_timing: id  0 | task 5533 | n_gen =   6649, tg =  70.88 t/s, tg_3s =  70.09 t/s
8.30.523.853 I slot print_timing: id  0 | task 5533 | n_gen =   6841, tg =  70.64 t/s, tg_3s =  63.16 t/s
8.33.532.948 I slot print_timing: id  0 | task 5533 | n_gen =   7072, tg =  70.82 t/s, tg_3s =  76.77 t/s
8.36.584.957 I slot print_timing: id  0 | task 5533 | n_gen =   7292, tg =  70.86 t/s, tg_3s =  72.08 t/s
8.39.601.591 I slot print_timing: id  0 | task 5533 | n_gen =   7536, tg =  71.14 t/s, tg_3s =  80.88 t/s

around 55-60t/s for generating reasoning, and ~65 for prose.

Total combined server output (mixed reasoning/code generation)

8.48.905.259 I slot print_timing: id  0 | task 5533 | prompt eval time =     792.44 ms /    28 tokens (   28.30 ms per token,    35.33 tokens per second)
8.48.905.262 I slot print_timing: id  0 | task 5533 |        eval time =  115216.01 ms /  8021 tokens (   14.37 ms per token,    69.61 tokens per second)
8.48.905.262 I slot print_timing: id  0 | task 5533 |       total time =  116008.45 ms /  8049 tokens
8.48.905.262 I slot print_timing: id  0 | task 5533 |    graphs reused =       9077
8.48.905.265 I slot print_timing: id  0 | task 5533 | draft acceptance = 0.95131 ( 5939 accepted /  6243 generated), mean len =  3.85
8.48.905.276 I slot print_timing: id  0 | task 5533 |   MoE arena = 0.9545 (51023812 hit / 53458080 reaches this turn), arena 38854.7 MiB

This is better than I had ever imagined!

[briansp2020]

Thanks for r22–r24 and the detailed write-up with logs. The movable-boundary slab is a really neat design. Sharing some field data in case it's useful; nothing urgent on our side.

Setup: 2 × R9700 (32 GB each), PCIe 5.0 x8 per card, 59 GB host RAM. Flash-Next GSQ-RCO IQ3_S and IQ3_XXS, -sm tensor, -c 262144, MTP n-max 3. The build is r24 (tree 1a580f937, matches release.json) plus a few local patches.

Your configuration (-ncmoe 48) works out of the box with defaults, the PLE table in VRAM:

	IQ3_S	IQ3_XXS
arena	37.3 GB (77.7%)	37.6 GB (92.0%)
arena hit	0.95 → 0.995	0.95 → 0.998
decode, greedy	92 t/s	107 t/s
decode, code	99–104 t/s	108–116 t/s

Prefill on a ~61K prompt scales with -ub as you'd expect from the per-ubatch staging: 1,485 t/s at 2048, 2,023 at 4096, and 2,201 at 6144, against 2,700 with everything in VRAM. 0 faults throughout. (--load-mode none ran out of host RAM on this 59 GB box for IQ3_S; the default load mode was fine, as your startup warning suggests.)

Two things I noticed, in case they help:

1. At -ub 2048 only, every prefill after the first is ~20% slower. The server's per-batch timing goes from 1.32 s to 1.72 s per 2048 tokens, and stays there. I couldn't pin it down. It persists with LLAMA_DROP_COMPUTE_BUFFERS=0, MOE_EXPERT_CACHE_PREFILL_SEED=0, GGML_CUDA_SLAB_HEADROOM_MIB=8192 (no slab extend), and -cram 0, and it isn't disk paging. Under rocprofv3 it shows up instead as one ~6.2 s staging copy on each GPU right at the start of the second prefill. It doesn't happen at -ub 4096/6144. Happy to run anything that would help narrow it down.
2. With partial offload (-ncmoe 14/18), the default GGML_CUDA_SLAB_RESERVE_MIB (8192) leaves too little for the slab on 32 GB cards. At N=18 it declines ("only 2880 MiB is available after the 8192 MiB reserve"); at N=14 the arena ends up at 0 MiB. Decode drops to ~40–47 t/s, against ~88 on r20. Following your error message's hint, GGML_CUDA_SLAB_RESERVE_MIB=4608 gives 83–87 t/s.

I also checked greedy purity on the host-expert paths: plain == draft-mtp holds on every path (cache, cache-off, all-VRAM). Outputs differ between paths only at near-ties (e.g. 0.4745 vs 0.4718), which reads as the expected rounding difference, not a bug.

Thanks again!

[stew675]

Thank you for the feedback. One thing I'd like to do is for the bits of the weights that MUST be VRAM resident that we release these from system RAM after they're loaded into VRAM. That will help with yoiur memory tightness somewhat.

Another possibility is to implement an expert RAM cache as well. Basically two-staged caching. Hot weights can be on GPU. Warm weights in RAM. Cold weights on SSD, and stream the SSD-based cold wegihts in on demand. This would allow people in your situation with limited RAM to even run Q4-sized model weights

[stew675]

This is now in as r28. --host-experts <pinned|mmap|auto> is a first-class option (previously only the LLAMA_MMAP_HOST_EXPERTS env var existed), folded into block 06, and llama-bench carries the flag too.

--host-experts mmap keeps the host-resident expert master in the pageable model mapping (CPU_Mapped) instead of the pinned ROCm_Host buffer, so those pages are reclaimable rather than non-swappable RAM. The default is unchanged (pinned).

Measured on 1x R9700, Qwen3.6-35B-A3B UD-Q4_K_M, -ncmoe 40, expert cache auto:

	prefill	decode	peak RSS	RssShmem
pinned	398.8 t/s	87.5 t/s	40.7 GB	19.0 GB
mmap	376.8 t/s	87.1 t/s	22.2 GB	0.36 GB

So about 5 percent slower prefill and no real decode change, for about 18.6 GB less non-swappable host memory. Greedy text is byte-identical between the two modes.

One caveat: on a box with enough RAM the model stays in page cache either way, so this does not reduce total resident memory by itself. What it changes is reclaimability (the pinned buffer cannot be evicted, page cache can) and the load-time peak. It matters most when the model, or the working set, sits close to or above what the kernel will keep cached, which is the direction of the two-stage (GPU / RAM / SSD) idea. The SSD streaming half is not in yet.

Release: https://github.com/stew675/llama-cpp-rdna-boosts/releases/tag/v16-a55e952b8-r28

[stew675]

These are my numbers regarding different -ncmoe values with IQ3_XXS. The size of the prompt here was 21K. ubatch is 2048

The true fix for this is to make the --fit algorithm be more slab aware, but this is difficult because the way that llama.cpp works is that it doesn't know the size of what's needed until we actually start to prefill.

I've generally found that it's best to just use -ncmoe 48 with a ubatch of 4096.

I do agree that at present it's possible to tie the algorithm into knots. I have a --fit campaign in wip/fit-slab-accounting that I've only partially done. More details in the README.md of that directory. I'll need to pick this work back up again soon.

[stew675]

r35: an optional bounded host tier for -ncmoe / -cmoe (--host-experts pool)

A follow-up to the -ncmoe work in this thread. Release v16-a55e952b8-r35 adds an optional managed host cache for host-resident experts, folded into block 06 (plus a kernel hardening in block 13).

For people running models larger than VRAM

The -ncmoe / -cmoe host experts are backed by a pinned (ROCm_Host) master. That is great when the host set fits comfortably, but it is non-swappable RAM. r35 adds a bounded alternative:

• --host-experts pinned (the default) and auto: unchanged, the usual pinned master.
• --host-experts pool: a bounded, pinned host pool, filled from the GGUF through the page cache, with its own budget and eviction.

The default stays pinned, so nothing changes for the fully-resident case. The pool is the option for limited-RAM systems.

Sizing it

• MOE_HOST_POOL_MIB is the process-wide budget. Unset or auto = MOE_HOST_POOL_FRAC (default 25) percent of the MoE host expert bytes (moe_host_expert_bytes), not the whole model. For 35B-A3B Q4_K_M -ncmoe 40 that is about 4645 MiB.
• N = an explicit MiB value, 0 = disabled.
• It is worth knowing that a small pool churns. On the long-prompt sweep (-n 3000, MOE_EXPERT_CACHE_MIB=2048), prefill on / off: 2048 MiB 42.5 / 44.0, 8192 43.3 / 43.9, 16384 43.9 / 43.9, pool off 44.7 t/s. Because of that the routing prefill is off by default (MOE_HOST_POOL_PREFETCH=1 opts in), and the default auto size (25 percent) lands close to the no-pool number.

Example

HIP_VISIBLE_DEVICES=0,1,2 \
llama-server \
  -m  <model gguf> \
  -ngl 99 \
  -sm tensor \
  -ncmoe 40 \
  --host-experts pool \
  -fa on \
  -t 8 \
  -c 16384 \
  --host 127.0.0.1 --port 8080

Add MOE_HOST_POOL_MIB=<MiB> to pin the budget, or MOE_HOST_POOL_MIB=0 to disable the pool again without dropping the flag.

Gates

On 3x R9700 (gfx1201), 35B-A3B Q4_K_M -ncmoe 40, both pinned and --host-experts pool produce the same text: short 359ff4337837 (-sm layer and -sm tensor), long-prefill 12d4fcd10886 (both splits), dense 4B 1c5d32ac537d, MUL_MAT_ID 931/931, prefill-logit KLD 0.000707 / 98.755 percent PASS.

Also in r35

• The parked slab peer-access fix (ggml_cuda_vmm_map_phys grants cuMemSetAccess to every peer device), which removes the --host-experts pinned + -sm tensor + partial-arena Page not present fault.
• The mul_mat_vec_q_moe row-tail clamp in block 13.

Since r15, briefly

• r33 removed --host-experts mmap (a no-XNACK GPU cannot read a pageable master in a kernel).
• r34 deferred block 12's ncclCommInitAll to the first large tensor, so decode-only -sm tensor no longer pays the eager NCCL init (-sm tensor -ncmoe 0 decode 21.6 to 69.5 t/s, output unchanged).

Release details: v16-a55e952b8-r35, base a55e952b8, canonical tip 645fd4989, net tree b2ba2bb32, strict 16/16 apply. Environment variables are documented in ENVIRONMENT.md section 1.2.1.

[stew675]

It's finally done! I'm sure there will be bugs to be found, but this now finally allows people with less VRAM than the model size is to effectively treat the system as a total of VRAM + RAM together, and if anything spills over that, it gets on-demand-swapped in from disk. I don't promise it will be fast if you push the system too hard.

Basically it works like this: VRAM Slab Cache (L1) <-> Host Pool (L2) <-> System Page Cache (L3) <-> HDD (L4)

The host pool, when enabled, default to 25% of the MoE expert weight size, but this can be tuned up/down. The smaller you make it, the more you're forcibly relying upon the system's page cache which may, or may not, work well.

What is does also allow though, is for people with big RAM to fully load their model weights into /tmp once, and then set a small Host Pool, and then you can rapidly restart/test server changes without worrying about your weights getting paged out.

This should allow for a lot of freedom to do crazy stuff now.