Expert caching that greatly increases performance. Self contained, off by default, use -ehs N to activate

[miltos22]

This has been closed with plans to organize and re-open

I have redesigned about half of the entire system in ways that fix all major issues exposed by this pr. I have not decided yet if ill push here or make a new clean pr with a new commit history after applying the final changes one by one. I still have some work but this is just a heads up that I am aware of the issues and im fixing them

Overview

I want to make this as short as possible. I made a CUDA only for now feature where it tracks a heat map of all expert usage and the hottest experts are cached on GPU and computed there, while cold experts are computed on the CPU. off by default can be set by "--expert-hot-s N" (or -ehs for short), autofit from free VRAM using the native auto fit functions sizing based on how many experts would be offset to the cpu "--expert-hot-s -1", or off (0, default). But if you want to test you an use --ecf to force it on

Also adds: hysteresis gate + dwell on slot swaps, per-expert scale handling, a CUDA-only backend guard (Vulkan/CPU are rejected with a warning as I have not managed to get non cuda gpus to work fully and honestly can't figure it out and it does nothing on a cpu install, but still left a force activate argument), and auto-activation of --cmoe when manual slots are selected as the GPU will act as caching not slots.

Measured on Qwen3.6-35B-A3B Q2_M / Q5_K_P with 8 GB VRAM: ~1.7-2.1x
steady-state decode speedup over stock (56 vs 33 tok/s, 36 vs 17 tok/s) using autofit.

Additional information

Known limitations:

• Tier only engages at n_tokens == 1 (single-token decode); multi-slotbatches fall back to stock mul_mat_id.
• Verified on separate gate/up expert tensors (Qwen-style). Gemma 4 seemed broken as it resulted in incoherent output and initially i theorized it was because of gate_up_exps models breaking my implementation. but aparently the model i tried to run had stopped working on the latest upstream too, even tho it works fine on the one from a few days ago. So does not seem like my issue. I am almost certain it works properly, but i would appreciate a correction
• CUDA-only by design; the guard logs a warning and skips the tier on other back-ends unless forcefully activated.
• Due to differences in how experts are processed on the CPU vs the GPU, even at temp 0 when different experts are cached, the output may slightly vary. I have run extended tests and confirmed this is not silent corruption and happens on the original llama too if you change "--n-cpu-moe"
• Not intended for multiple GPUs. Aparently it CAN work but I have no way of testing and developing it currently.

Requirements

• I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master CONTRIBUTING.md)
• AI usage disclosure: Yes. On most the steps I got stuck I brainstormed with AI on how to resolve each problem, although all final code was written or verified by me (Although note i am not heavily experienced). Ai also caught and corrected a few of the smaller bugs in my code or changes the defaults/names of variables when i could not be bothered to do it manually.

I am not to my knowledge, but thanks for the warning?

Some results in more detail

Some numbers have been updated to reflect the minor optimization I commited. The full optimization numbers are better, but I wont put them here till its ready as thats misleading

Original llama.cpp vs my pull request

Model	Size	S	Stock (tok/s)	Request (tok/s)	Ratio	Coherent after 128k tokens?
Qwen3.6-35B-A3B (Q2_M)	11 GB	126	33.25	57.2	1.72x	Fully
Qwen3.6-35B-A3B (Q5_K_P)	28 GB	43	17.34	35.93	2.07x	Fully
Qwen3.5-122B-A10B (IQ2_M)	29 GB	23	4.72	3.97	0.84x	Fully
Qwen3.5-122B-A10B (IQ2_M) (dwell=16)	29 GB	23	4.72	5.7112	1.21x	Fully
Laguna-S-2.1 (IQ3_XXS)	44 GB	17	1.35	1.79	1.33x	Fully
Gemma-4-26B-A4B (Q5_K_S)	18 GB	32	16.2*	48.4*	-	Both original and Mine failed. Seemingly identical.

• Gemma excluded: My downloaded model seems to degenerate even on original. Not a valid perf comparison. however when regenerating repetitive text this obviously gives a bigger bonus and that may have a few genuine uses.
  *It caues some CPU overhead, recommended to not run with -t at your max logical threads or if cpu usage is max during normal generation. Results on different hardware may vary

Important Arguments:
-ehs N | -1 - auto requires auto fit. | 0 - Disabled (default) | N - set manually, auto sets cmoe too |
--ecf | Forces slot caching to activate on backends i have not verified

[miltos22]

Guys im stuck. basically this tradeoff: small self contained diff similar in size to current, -10% best case performance compared to current push, + 60% In my testing with artificial bottlenecks worst case performance compared to current push, corruption edge cases mathematically impossible on all backends, batching disabled so slower reads (but still workable) and only -np1.

100 lines of extra code outside my files and around 500 within mine iirc that doesn't affect behavior without my flags and an extra op, +30% top end performance compared to current commit, +60% low end performance, batching works, sidecar, can use any -np N. Touches cuda kernel (when my flags off no changes)

Both: Multi gpu support (although still non optimized), dont affect process path if flags disabled, corruption edge cases mathematically impossible on all back ends. genuinely uses vram as additional memory rather than just a copy, freeing ram. but the first ones way is more hacky when the second ones is proper integration

I made both, but i am not sure which one is right.

I have made both because I was unsure if the tradeoffs were or not worth it. Which one do I push to this PR?

[siganos]

I hope there can be a way to limit this overhead.

Also i would appreaciate if you tried https://github.com/miltos22/llama-wackMall And told me if that fixes the issue. I am 99% sure it does. Just a heads up the arguments there are a bit different so read the readme

I did try the wackMall version and the behavior was similar in initial memory consumption, but i was getting a bit worse performance. I run it twice so that there was a sidecar file, but i believe it failed, following is the debug output

load_sidecar: loaded sidecar unsloth/DeepSeek-V4-Flash-0731-GGUF/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf.tier (fingerprint 6dc5e3b91c09a9fe)
tier_pread_kat: KAT pread failed
init: pread disabled (KAT failed)
init: AUTO-FIT ENGINE -> set S = 60 (Free VRAM: 21.86 / 31.40 GB, per-slot size: 360.69 MB, total experts: 256)
init: page hints: on
init: expert tiering on: 60 slots/layer, 129 tensors, 21.49 GiB pinned, seed coverage 71.2%

[miltos22]

I did try the wackMall version and the behavior was similar in initial memory consumption, but i was getting a bit worse performance. I run it twice so that there was a sidecar file, but i believe it failed, following is the debug output

Funny story. It needs special arguments to work properly. Iirc for the current release that would be: Don't use no mmap and instead set tam pool in GB. I think it's in GB? May be in slots don't recall. I'm finally going to sleep I've been 40 hours awake. I should have better documentation because without the rampool argument it's basically had no advantage. But my new push here when I actually push it is not going to need special treatment, no mmap will work fine without eating ram that can fit in gpu

[voidpush]

Both: Multi gpu support (although still non optimized), dont affect process path if flags disabled, corruption edge cases mathematically impossible on all back ends. genuinely uses vram as additional memory rather than just a copy, freeing ram. but the first ones way is more hacky when the second ones is proper integration

I have made both because I was unsure if the tradeoffs were or not worth it. Which one do I push to this PR?

Do the proper integration, but...
Maybe create an issue that properly explains the whole architecture so that maintainers can reason about the changes more easily, and use a phased implementation by breaking this PR into a few modular PRs that build upon each other.
Keeping in mind that the chosen integration should ideally make porting this feature to other backends relatively straightforward.

Good luck and keep up the good work. :)

[vlascik]

Please also make it work with multiple cards, it's kind of essential given the current hardware situation. Also, a lot of people will be running wildly mismatched GPUs - different VRAM sizes, mem bandwidths, TFLOPS, so, maybe a way to set priorities for the device placement would be a win? E.g. -ehs-prio CUDA1,CUDA0,CUDA3 ?

Architecture looks solid. Good luck, let us know if you need to test something.

[miltos22]

E.g. -ehs-prio CUDA1,CUDA0,CUDA3 ?

that can be achieved though existing variables. you can -ot the dense layers to the best gpu, and this integration will respect -ts, so if you will just be able to set less experts to- wait you dont want less experts you which will have the most used. OK! that should be done.

[miltos22]

As i was doing that i stripped all the tiering and swapping logic to try a rewrite for efficiency as i figured out it had some inefficiencies

I get why people did not attempt this. my test results alone, which just include mostly the run commands, 256 token runs along with the output, the prompt, the speed and a few lines of my own diagnostics alone are enough to where i can no longer give the data to an ai with 1 million context to summarize, as they dont fit. Now someone more knowledgeable OR more reckless could probably do this with less tests but i have to test different hypothesis to find the issues as I cannot understand the exact operation of the entire token generation loop just though code

[blakemartz]

thanks for your work on this. been trying out a lot of different approaches from different PRs to optimize performance on my machine. here are some copy paste test results from [AGENT]

Datapoint that may matter for the rewrite: I tested this on a load-balanced many-expert MoE (DeepSeek-V4-Flash-0731, 256 experts / 6+1 active, experts on CPU, 2× RTX 4090) and the routing distribution turns out to be the limiting factor, not the implementation.

• Slot economics on this model: one slot = 548 MiB across all layers, so 24 GB VRAM affords S≈29 of 256 (~11%).
• After a ~1.4K-token warm pass, the heat map shows 209–229 of 256 experts warm per layer — DeepSeek's aux-loss-free load balancing makes routing near-uniform, so a top-S cache has ~11% theoretical hit rate at this VRAM:model ratio.
• Measured steady-state decode: 12.77 t/s vs 13.45 for a static "pin 4 complete layers' experts to GPU" baseline using similar VRAM — the static pin wins because complete layers hit 100% by construction with zero LUT/swap overhead.

So the 1.7–2.1× from the Qwen A3B results likely depends on routing skew and a high slot:expert ratio; on balanced-routing models with hundreds of experts, heat caching can't beat static layer pinning. Might be worth a note in the docs about which model classes benefit.

Two ops footguns from testing (current branch, may be moot post-rewrite): -ehs -1 autofit silently resolves to 0 slots when -ngl is set explicitly (fit aborts → no cache, no warning), and manual -ehs N with that same fit-abort crashes during load unless -fit off is passed.

[miltos22]

* Measured steady-state decode: 12.77 t/s vs 13.45 for a static "pin 4 complete layers' experts to GPU" baseline using similar VRAM — the static pin wins because complete layers hit 100% by construction with zero LUT/swap overhead.

Thanks for testing. I have a few points i want to clarify, I will take into account the extra data

This is supposed to help the most when at least 40% of the models total fits in your vram. on other instances, especially ram limited, the current implimentation has a slowdown as the experts are only copied to the GPU not stored there, meaning you need N gpu experts size more ram to fit it in your ram. even if using mmap, it means higher cache misses on the mmap itself which is what causes the biggest stall. I hope this helps clarify what this implimentation does, however, soon it will actually store experts on the gpu, it already does so on my pc it just hasnt been pushed yet as im still working on it and its in a semi broken state with stuff stripped out for testing.

Furthermore, there is no proper multi GPU support in this version. It happening to not error out is a coincidence but the speed regression is major

The other parts i have also fixed. ALTHOUGH autofit not working without proper autofit was intentional too many people disliked it

Also I think you may be wrong about the hit rate hypothesis. its not at all uniform. as even if the majority of experts gets activated often, they do so at different frequencies. Someone else here already tested deepseek flash with what seems to be a slightly better hardware/quantization combination as they had higher speeds from the start and they got x1.7 or so speed

Also sorry for the caps i feel like i sound rude im just tired

[miltos22]

Im reffering to this

@miltos22, Thanks for you effort in doing this, the idea makes sense.

I tested it with deepseek flash 0731 UD-IQ3_XXS. I have a 5090 with 96G ddr5-6400, so memory is tight (important for my feedback). I checked with a few very simple queries like check the scripts that start the llama server. With vanilla llama i am getting 22-25 tps, while with yours i get 32-35 tps a very nice boost, but i have to wait a few minutes for the first token.

The problem is that your code uses significantly more memory than vanilla llama. With vanilla llama i can fit 160k context and still have ~10G unused memory in the system, while running your code i run out of memory and have to use mmap and ended up streaming from disk. I tried different parameters for ehs but it was always using more memory than i had available.

I hope there can be a way to limit this overhead.

[miltos22]

Sorry for delaying the push. Here is what ive been doing: Rebuilding everything I had wrote from scratch, trying to match on every level the equivalent scenario to stock as much as possible. For instance what my goal is: If all experts are calculated on the cpu using my own flag dependent path, they match stock with -cmoe. It was 25% behind. If the equivalent vrams worth of slots are used as the equevilent -ncmoe, it should match stock. (it was 10%ish too even with all experts on the GPU)
Initially I framed these as worthy tradeoffs for the simplicity. I am still optimizing this part. these are my goals

Unavoidable overheads and why they are fine:
Pcie bandwidth. Moving experts back and forth can be a VERY significant slowdown.

Why its fine:
My implementation only moves 1 expert per layer at a time, and how often it does that dynamically adjusts itself. It should always (or in the vast majority of cases) throttle the expert swap rate to not drop performance too much from overly swapping while still swapping a lot when it would benefit the overall generation.

GPU work: Currently all ways I tried to get the gpu to not calculate at all extra experts have failed. My workaround is having the GPU make calculations even for experts that are not present.

Why its fine
The GPU has a ton of free processing power when the entire model doesnt fit on the GPU, as its constantly waiting on the CPU, and its more often than not, constrained due to the sequentiality of operations ather than compute.
Speed loss can still occur, but only at high -np values is it a real deficit, which llama is already not heavily intended for

2 problems i faced and I want to give details on:

Moving experts back and forth, allows full use of ram + vram rather than just ram which was the old copy cache system. BUT it basically doubles bandwidth requirement.
How i resolved: Include both copy mode and move mode. autofit chooses based on available ram overhead, or you choose manually. If move is enabled, speeds are slower and the moving rate is halved, but the vrams worth of GB you save on your ram should provide a bigger benefit if ram constrained as that was the bottleneck compared to stock, so performance can only go up

Synchronize stalls: Due to the very nature of moving experts around, you need to synchronize to make sure you wont read an expert at a wrong state for instance. But doing that can actually have a pretty big cost

Solution: 2 different ones. not decided fully yet
1: expert hash verification. Send expert to gpu without any synchronization and after its verified to be there start actually calculating it there.
2: switch in and out constantly but only change where the experts are calculated every N tokens, after a synchronization with all the copies. BUT on move this makes it need an extra buffer on the CPU. The gpu is still fine because if you do this order operation you eliminate the need for a gpu buffer:

Move to CPU, sync, start calculating there and free gpu, that slot becomes eligible for copying GPU experts to it. move to GPU, synchronise, delete from cpu. And basically that loop happens over N tokens based on the auto adjust.

Lastly the proper implementation path which was what people seem to prefer faces a lot of barriers in regards to gpu back end compatibility. While my more "hacky" fixes worked for the simple implementation, they had tradeoffs, and the proper way is to forego them and fix the issues that caused them, and thats what ive spent most my time trying to do

These are obviously not all the challenges I faced working on these, simply some of the ones that are the easiest to talk about

Current numbers of custom cold expert CPU path vs -cmoe (both calculate all experts fully on the cpu):
First number stock second mine. This is where im trying to match stock not exceed it. just while allowing for dynamic expert splitting where stock doesn't averaged across 5 runs each
Tiny phi: 39 vs 37
q2 qwen: 20.65 vs 20.45
q4 qwen: 14.23 vs 14.19

So basically I am about as efficient as stock. meaning any benefit from caching built on top of this will now be pure gain. my previous baseline was 20-30% bellow stock so the gain was there just because it outperformed the loss

Currently working on: Trying a new idea for a more efficient way to calculate on the gpu

I have finished all the core systems redesign.

My final static gpu results are these, and i think they are as good as they can be given the complexity of splitting experts in llama without a core rework:

stock ncmoe 17average across 15 runs: 39.9
tier S=151 (equivalent vram) average across 15 runs: 37.1

Note this is not with expert caching, just static experts like stock just split per expert instead of per layer. I will now re introduce caching and i will likely be done in a day or two as of this edit.

I finally found why in my old build, that instance of multiplying 0s didn't return 0. I had forgot to have it clear a buffer, making it work on first launch and fail in subsequent which i had interpreted as a race condition incorrectly

Adding edge case logic failures such good autofit, warnings etc as well as efficient device to device transfers killing the pcie bottleneck on multi gpu that has been the biggest issue for people here. this is getting close to done

[mgabor3141]

[AGENT]

Tested on 4x RTX 3090 (96 GiB total) with unsloth's DeepSeek-V4-Flash-0731 UD-Q8_K_XL, 150.75 GiB, 43 layers, 256 experts, top 6.

-ehs -1 silently resolves to 0. It is only resolved inside the params.fit_params block in common/common.cpp, so if anything has already disabled the fitter it falls through to expert_hot_s = 0 and the cache is off. Both -ngl 999 and --n-cpu-moe N disable the fitter, and those are the natural way to run a spilling MoE, so this is easy to hit. There is also no log line to say so, because the expert_hotstore->log() call is itself guarded by expert_hot_s != 0. I ran a complete A/B before noticing both arms were byte identical. Warning instead of quietly falling back to 0 would have saved me a couple of hours.

The store only ever allocates on card 0. allocate() takes a single ggml_backend_buffer_type_t, and the caller in llama-context.cpp breaks after the first non-CPU backend. Meanwhile the autofit in common/fit.cpp derives S from final_gpu_model summed across all devices, so on multi-GPU the sizing and the allocation disagree by roughly the device count:

allocate: not enough memory to allocate the GPU hot store of 135 slots
(74562 MiB needed, 20770 MiB free on CUDA0)

Handing the cache a whole card with -ngl 999 --cpu-moe --tensor-split 0,1,1,1 -ehs 36 does run, but leaves about 59 of 96 GiB of VRAM idle with only GPU0 doing work, so it competes with plain --n-cpu-moe rather than complementing it. Sharding the store per device (split-mode layer has already assigned layers to devices by that point), or clamping S to the one device it actually allocates on, would fix the mismatch.

No perf number from me. -ehs requires all experts host-side, which is ~133 GiB for this model against 125 GiB of RAM here, so it was paging off a SATA SSD, and decode was still climbing when I stopped. Happy to test a multi-device version if you write one.

[miltos22]

I think I will soon close this pull request and open a new one. the scope has expanded into something thats both bigger and more refined, while truly adding multiple tiers of cache to the point it outperforms stock on large models by a considerable margin

[Green-Sky]

I think I will soon close this pull request and open a new one. the scope has expanded into something thats both bigger and more refined, while truly adding multiple tiers of cache to the point it outperforms stock on large models by a considerable margin

Mark this pr as draft then.

[miltos22]

multi gp

I can test the multi-gpu portion to see if it at least works. As far as getting benchmark numbers, that might produce inconsistent number. My second GPU is limited by bandwidth and it is weak so I currently use it as VRAM extension. I already tried your commit from some 8 hours ago but I had to shift a lot of layers into the weak GPU to get it to work so the benchmark numbers are essentially useless.

Hi, would you be able to test with me before I push anything or make a new pull request encompassing everything? Im not pushing it here so lets coordinate

[miltos22]

I think I will soon close this pull request and open a new one. the scope has expanded into something thats both bigger and more refined, while truly adding multiple tiers of cache to the point it outperforms stock on large models by a considerable margin

Mark this pr as draft then.

Thanks for the heads up

[Tha14]

multi gp

I can test the multi-gpu portion to see if it at least works. As far as getting benchmark numbers, that might produce inconsistent number. My second GPU is limited by bandwidth and it is weak so I currently use it as VRAM extension. I already tried your commit from some 8 hours ago but I had to shift a lot of layers into the weak GPU to get it to work so the benchmark numbers are essentially useless.

Hi, would you be able to test with me before I push anything or make a new pull request encompassing everything? Im not pushing it here so lets coordinate

I've sent you an email 😃

[rlewczuk]

I've got a few boxes I can test it (including ROCm ones, not sure if applicable), would love to see deepseek-v4-flash running on them fast.
Looked at https://github.com/miltos22/llama-wackMall and I see it is 3 days old.
Is this the latest version (one to be tested) ?

[miltos22]

I've got a few boxes I can test it (including ROCm ones, not sure if applicable), would love to see deepseek-v4-flash running on them fast. Looked at https://github.com/miltos22/llama-wackMall and I see it is 3 days old. Is this the latest version (one to be tested) ?

It is not the latest. In fact I have not worked on that for about 2 weeks. I only updated it 3 days ago in case people wanted to test the rampool to the latest not stripped for testing build i had at hand (which was another week old) So It is pretty slow compared to the actual latest wackMall research repo at my own pc as it lacks semantic heatmap database. it also needs you to understand the flags well to run it, they are well documented but doing something wrong like using -no-mmap insttead of the rampool flag would be detrimental as the rampool is the tiered version of no mmap already. Finally if im not mistaken that one has even worse gpu support than the one here. But its comparitive feature wise to what the merge request will have just a lot more hacky and less polished

llama-wackmall is a research repo where the original idea came from. Not a release grade repo. It may be obvious by reading the chat but I am a bit of a perfectionist so i wont release the latest version of llama-wackMall, with the semantic heatmap and prediction untill it is at a satisfactory state. And currently im not trying to get it to be in such a state as im putting all my programming time into refining the pull request version. I will probably have it up within the day in the same repo of this pull request

[rlewczuk]

Ok, then I'm waiting until you let us know it's ready for testing. No rushing though, just waiting for signal.
Thanks for this great work, it might be game changer for a lot of folks (including me).

[miltos22]

@rlewczuk @Tha14 @mgabor3141('s agent?) @blakemartz @vlascik @clort81 @siganos

I have added the features and improvements you requested. Would you like to test? I am closing this and opening a new request. I hope you guys see as much of a gain as I do.

[miltos22]

Im reopening with a broader scope

[Tha14]

@miltos22 Try to ping me in the new PR and if you can't just send me an email please.

[miltos22]

@miltos22 Try to ping me in the new PR and if you can't just send me an email please.

Its already up, just not merge request yet cause im writing it

[miltos22]

@Tha14 I can't tag you in the other pr

@rlewczuk it's ready

[Pix13]

Looks like it's the same concept as https://github.com/borisk1/llama.cpp-fusion fork with its moe-cache feature

[Atomic-Germ]

@Pix13

Looks like it's the same concept as https://github.com/borisk1/llama.cpp-fusion fork with its moe-cache feature

And Atomic-Germ/Guanaco... which was the first of these. It's still not decided how the community will integrate that feature but it's pretty clear it will. I've been the Guanaco extension for months, but it's never up to date with the mainline llama.cpp because it changes so fast.

Hopefully they figure out that gguf's already structured in a predictable way you can just address and then grab the active experts for vram, and the less frequent but recently used ones in system ram to a configurable number, and the rest on disk. Take the moe concept and lean in hard, that's where you logically arrive.