Generic Hybrid MoE

Generic CPU/GPU MoE execution

Experimental hybrid execution is available on moe-cache at 28d73c87c. It keeps frequently used experts on CUDA and executes selected cache misses on a persistent CPU pool while other misses transfer to CUDA.

Normal GPU execution remains the default. Hybrid mode is opt-in. Eligibility comes from model operators, tensor layout and backend capabilities, without a model-name whitelist. This is broader design coverage than the models qualified below; it does not certify every MoE.

Navigation: Exact commands and assets | Ordinary cache setup | Host pinning | Source guide

The current CLI/profile update simplifies configuration. The performance tables retain their original source checkpoints; they are not measurements of this update.

Development status - October 10, 2026

The latest scheduling work is saved in a local, unpublished research checkpoint. The build and launch commands below still describe the published source checkpoint linked above. They do not activate the new experimental prepared-MTP controller.

The retained changes reduce repeated readiness queries and backend fences, share the existing CPU service between independently owned prepared schedules, and allow known inputs to wake the producer earlier. The experimental controller uses the supplied model's MTP head and upstream graph/arithmetic; it changes execution scheduling and intermediate-value ownership. The ordinary controller and normal GPU execution remain available. The new controller stays opt-in and is not a universally qualified replacement.

Latest development observation, Flash Next	Native decode
MTP3, p-min 0.5, shipped static profile, adaptation off	72.37 tok/s
MTP3, p-min 0.5, shipped static profile, asynchronous occurrence adaptation	Highest observed 92.71 tok/s

These used an 8,194-token real-text prompt, 2,048 generated tokens with EOS respected, 64 actual resident slots per layer, GPU miss fraction 0.17, 16 CPU participants, F16 KV, 16K context, 2,048-row batch/ubatch, PLE, both decode overlaps and backend sampling. The adaptive observation had different output/acceptance work from its paired control; it is not a repeat-proven controller-only gain. Do not interpret adaptation-off versus adaptation-on as a regression comparison or compare either directly with the historical table below.

A topology-only trial of the existing conditional graph path versus device-tail execution completed with exact own outputs, acceptance, routes and residency maps at 72.380 versus 72.373 tok/s. No useful serving gain was established, so the existing device-tail path is retained. Focused cancellation/deadline tests passed; diagnostic warm windows had small kernel-work differences and do not prove a causal per-phase speedup.

Flash Next and Ornith have focused native/serving evidence for the retained scheduling work. Complete shared-KV/chained-head coverage, other models, concurrent requests, physical multi-GPU and native Windows qualification remain pending for the new controller. Existing published model results below do not establish those new-controller guarantees. No public activation command is documented until that controller and its source are published.

Build and launch

git clone --branch moe-cache https://github.com/GenerelSchwerz/llama.cpp.git
cd llama.cpp
git checkout 28d73c87cb90a9b78d4357ff1b456f456bfa1164
cmake -S . -B build -DGGML_CUDA=ON -DLLAMA_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server test-llama-archs test-moe-cache test-arg-parser -j18

Use the executable and shared libraries from the same build. The qualified implementation requires NVIDIA CUDA. This launch illustrates a 16K context with 64 expert slots; choose the budget for your model and GPU.

./build/bin/llama-server -m /path/to/model.gguf --moe-hybrid on \
  -ngl all -fa on -c 16384 -b 2048 -ub 2048 -np 1 -t 16 \
  --moe-expert-cache-size 64 --moe-expert-cache-host-pinned-mb 0 \
  --load-mode none --lazy-mode on --fit off \
  --decode-overlap --decode-boundary-overlap --ple-prefetch --phase-aware-workspace

Look for provider=source-core, completion/publication counters and zero errors in the log. --moe-hybrid on preserves checked failure behavior when execution cannot qualify. No hybrid environment variables are required; shared overlap and the source CPU allocation policy are configured automatically. --moe-hybrid off overrides inherited activation and selects ordinary execution. An enabled flag does not prove that an overlap or fusion ran.

Choose the controls

Control	Meaning
--moe-expert-cache-size N	Resident expert slots per cache owner. Not a GPU percentage.
--moe-expert-cache-mib MiB	Alternative explicit byte budget; do not combine with slot mode.
--moe-gpu-miss-fraction 0.17	Optional hardware tuning, default 0.17. Assign about 17% of distinct cache misses to GPU transfer, at 1/256 resolution. Remaining misses use CPU service. Hits already run on CUDA.
-t N	CPU participants. Tune against the actual core topology.
--load-mode none	Keep the host model source in RAM when it fits.
--moe-expert-cache-host-pinned-mb 0	Automatic registration with bounded staging fallback. It does not promise full pinning.
--moe-source-graph-capacity	Optional reusable shapes inferred from capacity. May change outputs; only affects source execution.
--backend-sampling	Optional target sampling through the backend.

Source mode permits ordinary prepared CPU operators whose allocation freedom has not been proven, including OpenMP runtime behavior. This is not a memory limit; resource ownership, capacity and completion checks still apply. In a models.ini preset use moe-hybrid = on; optionally tune moe-gpu-miss-fraction = 0.17. The selection is fixed at process startup and shared by contexts in that process. For Windows partial pinning, read the platform guide; native Windows hybrid generation has not been qualified here.

CUDA 12.3+ normally enables module lazy loading by default; older compatible installations can request CUDA_MODULE_LOADING=LAZY. This is separate from --lazy-mode on for model loading.

MTP and history matching

For a compatible external MTP head, add:

--spec-type draft-mtp --model-draft /path/to/mtp.gguf \
--spec-draft-n-max 3 --spec-draft-p-min 0.5 \
--spec-mtp-rs-planes 0 --spec-draft-ngl all \
--spec-draft-moe-expert-cache-size 0 --spec-draft-ubatch-size 2048

Omit --model-draft when the model supplies its supported embedded head, as in the Ornith recipe. Three drafts means up to three proposed tokens plus the pending actual token in target verification. Head precision, draft length and confidence threshold affect acceptance and speed.

An optional history chain can replace or extend drafts with matching recent token history:

--spec-lookup-chain 2 --spec-lookup-chain-min 3

This is bounded history matching integrated with the generic speculative path. It is not the complete Strata suffix drafter. Repetition can make it very effective; unrelated prompts may show no gain. Serving respects EOS in these tests.

Profiles and adaptation

Keep these three decisions separate:

Decision	Configuration
Offline expert statistics/rankings	--moe-expert-profile model-profile.gguf or a geometry-compatible ranked STRP file
Hardware placement and miss split	Slots/MiB, threads and GPU miss fraction
Online residency changes	--moe-profile-adapt occurrence for asynchronous occurrence adaptation; occurrence-sync is the synchronous control; default is off

Profiles use the existing grouped cache owner. They do not create another residency map. Model-bound GGUF statistics validate source identity and complete bank coverage. The collection command is a bounded diagnostic; held-out profile quality has not been established.

Memory across prefill and decode

Question	Generic execution at this checkpoint	Original Strata 99f3dbd
Automatic expert-cache sizing?	Explicit slots or MiB. Ordinary --fit does not maximize hybrid expert residency.	--expert-cache auto sizes from available VRAM after session resources and reserve.
Does prefill use cached ownership?	Supported cached prefill and source decode share the authoritative cache owner. Sequential prompt rows remain prompt work.	Its native runtime owns the expert arena and resident map.
Reclaim prompt workspace?	--phase-aware-workspace can shrink compute workspace. Bounded prefill staging keeps two 64 MiB lanes per owning CUDA context.	Can borrow an arena tail for prefill, then reload lent experts.
Grow decode cache into freed workspace?	No automatic lending or cache growth.	Prefill borrowing reuses resident-arena space.
Clear all experts at phase change?	No blanket reset on the qualified path; the published Flash run reported zero legacy/staged handoff resets.	Reloads the experts affected by its borrowing policy.

Sources: generic loader, workspace, cache, original Strata generator.

Comparison with original Strata

These are saved single-run observations on one RTX 5070 Ti 16 GB, approximately 62 GiB host RAM and 16 CPU participants. The baseline is the original Niko1221/Strata executable, pinned to 99f3dbd, tested October 3. No private port is used in this comparison.

Both used the same Flash Next GSQ-RCO IQ3_XXS target GGUF source, the same 8,194-token rendered dossier, a 16K context, F16 KV, 2,048-row prefill chunks, greedy generation, MTP with up to three drafts and a 0.5 confidence threshold. Both generated 2,048 tokens with EOS honored.

Recorded configuration	Generic public build, October 7	Original Strata, October 3
Decode, server/native final timing	87.82 tok/s	110.26 tok/s
Prefill	805.82 tok/s	1,764.81 tok/s
TTFT	10.192 s, HTTP first token	4.679 s, stdin first T record
Peak sampled VRAM	12,797 MiB	13,499 MiB
Resident expert/layer assignments	3,072: 64 x 48 layers	4,608: 96 x 48 layers
Profile/adaptation	Shipped ranked profile, occurrence adaptation	Uniform-96 ranked profile, adaptation every 4 steps/up to 96 swaps
Draft head	Local Q2 expert/F16 dense GGUF conversion	Native Q2 MTP pack
Additional drafting	History chain 2	Suffix drafting off
MTP accepted/proposed	1,351 / 1,595	1,216 / 1,564

The generic configuration is slower in this observation. These are not cache-budget-matched or quality-matched results. Strata's native pack also logged an experimental Q5_K output head; its dense/head execution is not identical to the GGUF runtime. Output parity is not asserted. TTFT methods differ, and prefill numbers are sensitive to disk/page-cache state. Do not attribute the difference to one kernel or to generic model support alone.

Execution design	Generic	Original Strata in this run
Integration	llama.cpp models, graphs, server, KV and sampler infrastructure	Native Flash Next runtime and packed expert/MTP files
Miss execution	Persistent CPU pool plus CUDA transfers, one grouped residency owner	Persistent CPU pool plus kernel-copy GPU service, native resident map
Graph/state handling	Capability checks, owned prepared schedules, optional capacity reuse, normal MTP rollback/state contracts	Native model schedule and native verification state
Serving	Existing llama.cpp API/server; request isolation and ordinary executor retained	Native --serve stdin protocol in this measurement
Other model evidence	See below	No same-model measurements collected for this table

Strata has other capabilities beyond this particular run. See its pinned source and documentation. Missing comparison measurements do not mean it cannot support a model.

Other tested models

These October 6-7 generic snapshots are included in the published checkpoint. Only the Flash row above was rerun on the final 081cf1d79 composition. All rates below are final server timing-log values, not output tokens divided by whole HTTP request time.

Model / configuration	Prompt / output tokens	Decode tok/s	Prefill tok/s	TTFT	Peak MiB
Ornith 1.5 35B A3B APEX MTP I Compact, embedded Q8 MTP3, history 2, calibrated profile/adaptation	8,194 / 2,048	104.34	2,603.99	3.164 s	13,099
Gemma 4 26B A4B Q4_0 + Q8 MTP3, technical dossier, no history	12,705 / 2,048	77.16	2,898.54	4.414 s	7,075
Same Gemma, repeated-text prompt + history 2	8,440 / 2,048	125.10	3,020.39	2.823 s	7,099
Nemotron 3.5 Lightning 30B A3B Q4_K_M, no MTP/profile/adaptation	8,425 / 2,048	105.00	2,946.40	2.872 s	15,211
DeepSeek Coder V2 Lite Instruct Q4_K_M, no MTP/profile/adaptation	13,198 / 256	3.22	388.11	34.040 s	8,233

Important distinctions:

• Nemotron: the matched ordinary cached GPU control reached 117.51 tok/s with the same peak VRAM. Hybrid is functional here, not faster. A separate 64K Q8 KV capacity check completed 64,512 real-text prompt tokens plus 64 output tokens, with 723 MiB minimum reported free memory. It is not a full-context throughput test.
• Ornith: 64K Q8 KV was configured, but this long-generation run used an 8K prompt. Near-64K fit was not proved. Earlier ordinary-cache results used another configuration and are not a matched control for this row.
• Gemma: repeated text produces unusually high draft/history acceptance. The 125.10 result is not a general prompt ceiling. Shared-KV MTP keeps its checked fallback where early draft overlap cannot qualify, even with both overlap flags set. The technical-dossier repair matched the healthy mixed-execution sampler control byte-for-byte over 2,048 output tokens.
• DeepSeek: generation completed, but 3.22 tok/s is still poor. A separate ngram-simple trial reached 3.05 tok/s, not a gain. DeepSeek V4.1 integration was explicitly excluded from this checkpoint.
• LFM2.5 8B A1B Q5_K_M: a routed-serving smoke completed 12 prompt/32 output tokens at 46.09 tok/s versus 111.27 on the ordinary path. This short test is not comparable to the long runs or a full numerical qualification.

What has and has not been validated

Covered by current evidence	Remaining limits
Original quantization routes, mixed CPU/GPU numerical controls, resource ownership, graph/state reuse and sampler binding fixtures	Full every-model coverage is not established
Flash, Ornith, Gemma, Nemotron and DeepSeek serving on Linux/single CUDA GPU	Native Windows hybrid and physical multi-GPU generic serving remain unqualified
Focused simultaneous/staggered two-sequence fixtures	Broad parallel throughput, cancellation and all MTP families remain unqualified
Profile loading/observation and asynchronous occurrence policy	Representative held-out generated-profile quality and live profile replacement remain unqualified

A four-sequence ordinary LFM 230M rollback fixture still fails on both the frozen control and candidate; this guide does not declare that path fixed. GPU-only arithmetic can differ from mixed CPU/GPU arithmetic. Validate against the route-matched numerical control before treating such differences as hybrid corruption.

See reproduction and machine-readable evidence. No new GPU benchmark was run to write this guide. Attribution for reused MIT components is retained in the repository README and license notices.