Skip to content

Generic Hybrid MoE Reproduction

Jump to bottom
Rocco A edited this page Oct 10, 2026 · 3 revisions

Generic hybrid: reproduction recipes

Source scope: these commands reproduce the published checkpoints and cases listed here. The October 10 development status concerns a separate unpublished scheduling checkpoint; these recipes do not enable its experimental prepared-MTP controller.

Usage and results | Source checkpoint | Original Strata checkpoint

These recipes expose the recorded server arguments, requests, profiles and benchmark environment. Replace file paths only for configuration reproduction. Different model revisions, heads, packs, drivers, core placement or inherited settings can change results; the supplied assets alone do not guarantee a matching speed or output.

Get the assets

git clone https://github.com/GenerelSchwerz/llama.cpp.wiki.git wiki
export ASSETS="$PWD/wiki/hybrid"
export REPO=/path/to/llama.cpp
export ENGINE="$REPO/build/bin"
File Contents
launch-arguments.json Complete generic argv and recorded environment, with path placeholders
run-case.py Prints the complete shell command; executes only with --execute
measurements.json Selected actual timings, outputs/hashes, cache counters and teardown evidence
MANIFEST.json SHA-256 and byte length for each asset
Asset notes Head/profile provenance, helper scope and MIT licenses

Model weights and native packs are not included. The Flash Q2-expert/F16-dense MTP GGUF was locally converted; a stock Q8 head is a valid different trial, not the same benchmark. Keep the public build and its libraries together.

CPU placement used in three cases

Flash, Ornith and repeated-Gemma measurements used a Linux-only interposer that pins generic CPU-pool workers to distinct physical cores inside the taskset mask. The recorded helper's older caller hook does not match the published pool-run ABI; the supplied source omits that inactive hook. Caller pinning is not claimed. It is an isolated benchmark helper, not an engine or production dependency. It does not contain the private specialized executor or its kernels. Its ABI is pinned to this source checkpoint; do not use it for concurrent contexts.

export AFFINITY="$PWD/benchmark-affinity.so"
flock /tmp/beellama-cuda-build.lock \
  c++ -std=c++17 -shared -fPIC \
  -I "$REPO/ggml/src/ggml-cpu" -I "$REPO/ggml/src" -I "$REPO/ggml/include" \
  "$ASSETS/benchmark-affinity.cpp" -ldl -pthread -o "$AFFINITY"

The recorded mask was 0-15; adapt it to your CPU as a new configuration. The Nemotron, technical-Gemma and DeepSeek cases did not use this helper. The quickstart omits it.

Exact generic launches

The runner reads the published argv/environment JSON, substitutes ENGINE, MODEL, MTP, ASSETS and AFFINITY, and prints the full command before launch. It clears inherited hybrid/graph-disable overrides. It does not choose cache sizes or silently change the case. No heavy profiler is enabled; the recorded light diagnostic settings remain explicit in the JSON.

Use one server at a time. The following lock order is the workstation's shared convention: build lock, then GPU lock. Send the request from another terminal, then stop the server with Ctrl-C and retain the log.

Flash Next GSQ-RCO IQ3_XXS, MTP3 and history 2

export MODEL=/path/to/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
export MTP=/path/to/matching-q2-expert-f16-dense-mtp.gguf
python3 "$ASSETS/run-case.py" flash                     # print only
flock /tmp/beellama-cuda-build.lock \
  flock /tmp/beellama-single-gpu.lock \
  python3 "$ASSETS/run-case.py" flash --execute 2>&1 | tee flash-server.log

For clarity, this is the complete server-argument portion for that case:

"$ENGINE/llama-server" --offline --model "$MODEL" --spec-draft-model "$MTP" \
  --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.5 \
  --spec-mtp-rs-planes 0 --spec-draft-ngl all \
  --spec-draft-moe-expert-cache-size 0 --spec-draft-ubatch-size 2048 \
  --alias qwen-gsq-iq3 --host 127.0.0.1 --port 24753 \
  -c 16384 -b 2048 -ub 2048 -np 1 -t 16 -ngl all -fa on -fit off \
  --load-mode none --lazy-mode on \
  --moe-expert-cache-size 64 --moe-expert-cache-host-pinned-mb 0 \
  --decode-overlap --decode-boundary-overlap --ple-prefetch --phase-aware-workspace \
  -ctk f16 -ctv f16 -kvo --cache-ram 0 --ctx-checkpoints 0 \
  --jinja --no-warmup --experimental-logs -n 2048 \
  --moe-source-graph-capacity --backend-sampling \
  --spec-lookup-chain 2 --spec-lookup-chain-min 3

The runner also sets GGML_MOE_EXPERT_PROFILE to flash-ranked-profile.bin and GGML_MOE_HYBRID_PROFILE_ADAPT=occurrence, along with the recorded source, copy/fusion and affinity settings. Running this argument block without its environment uses a different configuration.

Other models

Each command below expands to the complete launch in launch-arguments.json. Use the same ordered flock ... python3 ... --execute wrapper as above. Recipes retain measured context/cache choices; these are not universal recommendations.

# Ornith: MTP is embedded, no external MTP variable needed.
export MODEL=/path/to/Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf
python3 "$ASSETS/run-case.py" ornith

# Gemma technical dossier: external shared-KV Q8 head, no history chain.
export MODEL=/path/to/gemma-4-26B-A4B-it-Q4_0.gguf
export MTP=/path/to/mtp-gemma-4-26B-A4B-it-Q8_0.gguf
python3 "$ASSETS/run-case.py" gemma-long

# Same Gemma with repeated text and history chain 2.
python3 "$ASSETS/run-case.py" gemma-repeat

# Nemotron: no MTP, profile or adaptation.
export MODEL=/path/to/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_K_M.gguf
python3 "$ASSETS/run-case.py" nemotron

# DeepSeek Coder V2 Lite: no MTP, profile or adaptation.
export MODEL=/path/to/DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf
python3 "$ASSETS/run-case.py" deepseek
Case Context / KV Slots Drafting Profile/adaptation Port / request file
flash 16,384 / F16 64 MTP3 + history 2 Ranked Flash / occurrence 24753 / flash-request.json
ornith 65,536 / Q8_0 144 MTP3 + history 2 Ornith calibration / occurrence 24751 / ornith-request.json
gemma-long 16,384 / F16 32 MTP3 Off 8080 / gemma-long-request.json
gemma-repeat 16,384 / F16 32 MTP3 + history 2 Off 24751 / gemma-repeat-request.json
nemotron 65,536 / Q8_0 74 None Off 8080 / nemotron-request.json
deepseek 16,384 / F16 16 None Off 8080 / deepseek-request.json

All cases use GPU-miss fraction 0.17, 16 CPU participants, -b 2048 -ub 2048, one request, backend sampling, capacity reuse, both overlap flags, PLE prefetch and phase-aware workspace. A model's capabilities determine which overlaps actually qualify.

Exact requests and measurements

Once /health is healthy, send the saved JSON bytes to the native endpoint. For Flash:

curl --fail --silent --show-error http://127.0.0.1:24753/health
curl --fail --silent --show-error --no-buffer \
  -H 'Content-Type: application/json' \
  --data-binary "@$ASSETS/flash-request.json" \
  http://127.0.0.1:24753/completion | tee flash-response.sse

Change the port and request filename according to the table for other cases. Requests already contain the exact prompt, greedy sampling, seed 123, cache_prompt=false, ignore_eos=false, streaming tokens and per-token timings. DeepSeek requests 256 outputs; the other cases request 2,048. An EOS can end a future run sooner; report the actual count.

Prompt asset Recorded token counts
flash-dossier.txt Flash / Ornith: 8,194
repeat-rich.txt Flash: 8,655; Gemma: 8,440
technical-dossier.txt Gemma: 12,705; DeepSeek: 13,198
nemotron-dossier.txt Nemotron: 8,425

The prompts are already rendered text. Sending them through /v1/chat/completions adds templating and changes the comparison. Retain MIT notices when redistributing the documentation-based dossiers.

Use slot print_timing final eval speed for decode and prompt eval for prefill. HTTP total time includes prefill and cannot substitute for decode speed. Record client first-token time separately. The simple curl recipe retains the response but does not calculate TTFT; the published TTFT values came from timestamped benchmark clients.

For matching workstation evidence, also retain:

nvidia-smi --query-gpu=timestamp,memory.used,utilization.gpu \
  --format=csv -lms 100 > gpu.csv

Run this sampler in a separate terminal and stop it after teardown. Keep the expanded command, exact source/build identity, request/response, final timing log, cache/source completion counters and sampled peak VRAM. Check that the owned server exited, its port closed and its GPU allocations were released. Report output hashes/acceptance separately from speed. GPU-only versus hybrid arithmetic does not imply exact output equality.

Ordinary GPU control

Normal execution remains available. In a fresh shell, use the same model/context/KV/cache/threads and MTP head/sampling, set GGML_MOE_HYBRID=off, and omit the source-only capacity/profile/adaptation settings. Define history and backend-sampling choices explicitly. This is ordinary cached GPU execution, not necessarily full weight residency. Do not call different cache budgets a matched experiment.

Original Strata baseline

The comparison uses the original source at 99f3dbd0b21d1401b3769e0c0d963913607f380b, not this fork's private standalone port. See the original installation guide for matching expert/MTP packs. This recipe requires the recorded native IQ3_XXS expert pack and Q2 MTP pack; substituting another pack is another configuration.

The original engine used downloaded llama.cpp dependency sources matching 3cf03257f219afbe7334045ff7c6a06ac68c627d. All 1,376 tracked ggml files were checked against that commit when writing this guide:

git clone https://github.com/Niko1221/Strata.git strata-original
export STRATA="$PWD/strata-original"
git -C "$STRATA" checkout 99f3dbd0b21d1401b3769e0c0d963913607f380b
git clone https://github.com/ggml-org/llama.cpp.git strata-ggml
git -C strata-ggml checkout 3cf03257f219afbe7334045ff7c6a06ac68c627d
export STRATA_GGML="$PWD/strata-ggml"
flock /tmp/beellama-cuda-build.lock \
  cmake -S "$STRATA" -B "$STRATA/build" -G Ninja \
  -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF \
  -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_CUDA_COMPILER=/opt/cuda/bin/nvcc \
  -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
  -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache -DSTRATA_GGML_DIR="$STRATA_GGML"
flock /tmp/beellama-cuda-build.lock cmake --build "$STRATA/build" --target strata -j18

CUDA architecture 120 and /opt/cuda describe the recorded Blackwell/Linux build. Change them for another device/toolkit and report the change.

export PACK=/path/to/native-iq3_xxs-pack
export MTP_PACK=/path/to/native-q2-mtp-pack
export MODEL=/path/to/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
export MODEL_SECOND_SHARD=/path/to/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf
python3 "$ASSETS/run-case.py" original-strata
flock /tmp/beellama-cuda-build.lock \
  flock /tmp/beellama-single-gpu.lock \
  python3 "$ASSETS/run-case.py" original-strata --execute \
  > original-strata.stdout 2> original-strata.stderr

The runner expands this exact recorded command/environment and waits for READY, sends the saved GEN request, then sends QUIT after DONE:

CUDA_VISIBLE_DEVICES=0 CUDA_MODULE_LOADING=LAZY GGML_NO_BACKTRACE=1 \
LD_LIBRARY_PATH=/opt/cuda/lib64 \
nice -n 10 taskset -c 0-15 "$STRATA/build/strata" \
  --pack "$PACK" --native "$MODEL" --ple-gguf "$MODEL_SECOND_SHARD" \
  --expert-profile "$ASSETS/strata-uniform96.bin" --expert-cache 3460 \
  --prefill 2048 --spec 4 --spec-min-p 0.5 --max-context 16384 \
  --pool-workers 15 --prompt-cache 0 --suffix-draft 0 \
  --pcie-frac 0.17 --adapt-every 4 --adapt-swaps 96 --pcie-mode kernel \
  --kv fp16 --stop-eos --mtp "$MTP_PACK" --serve

--spec 4 includes the actual token, corresponding to up to three drafts. The request is GEN 2048 temperature=0 spec_min_p=0.5 <8194 comma-separated token IDs>, stored verbatim in strata-request.txt. Do not pipe QUIT before generation completes. The recorded INFO line reported 4,608 resident slots, despite the --expert-cache 3460 argument; compare effective resource logs, not requested values.

The recorded original executable SHA-256 was eac0ac51cde5d8c001dffb390c11d9c6a59f48022f170608fe9c725fa5200bf5. Compiler/library differences can change a rebuilt hash. Original Strata's TTFT was stdin-to-first-T, not HTTP TTFT. Its DONE record supplies actual prompt/output counts and PP/TG times. Its output-ID hash is not directly comparable to our text hash.

Collect and load a model profile

These commands use source 28d73c87c. Build it with the current quickstart; the historical benchmark commands above retain their recorded checkpoint and environment.

Generate a weighted corpus profile

Use the in-tree collector; no Python package or separate repository is required. Build test-llama-archs with the matching engine libraries. This example gives general and coding requests equal workload weight, independently of their output lengths:

{
  "version": 1,
  "weights": {"general": 0.5, "code": 0.5},
  "requests": [
    {"family": "general", "prompt_file": "general.txt", "max_tokens": 256},
    {"family": "code", "prompt_file": "code.txt", "max_tokens": 256}
  ]
}

Save this as corpus.json. Prompt file paths are relative to that file. Each request accepts either prompt_file or inline prompt. Use raw model input, including its chat template if required. Use representative calibration prompts and separate held-out prompts for evaluation.

./build/bin/test-llama-archs --collect-moe-profile model.gguf \
  --profile-corpus corpus.json --profile-output calibration.gguf \
  --profile-threads 16 --replay-cache-slots 64 --replay-load-mode none

The collector automatically enables generic hybrid execution; no activation environment is needed. Optionally tune --moe-gpu-miss-fraction F, default 0.17. Unset GGML_MOE_EXPERT_PROFILE for fresh collection. Choose slots, threads and miss fraction for your hardware; they are not properties learned into the profile. Collection loads one model and uses a fresh context for each serial request. Greedy generation stops at EOG or its requested limit. A request that ends before any observed decode row fails explicitly; the collector does not continue past EOG to invent observations.

The score for each original source/expert is:

request rate = measured expert count / actual target decode rows
workload rate = mean(request rates in that workload)
score = sum(normalized workload weight * workload rate)

This reproduces the normalized workload-mixture method. It does not give longer requests more weight. Raw counts remain separate from scores. Cache groups combine complete-bank scores using their existing expert payload-byte weights; equal scores prefer the lower expert ID. Profile generation does not encode a fixed cache size, PCIe split or model architecture.

The profile is metadata-only GGUF version 2 with optional source-indexed F64 moe.profile.ranking_scores. Raw version 1 profiles remain readable. The adjacent calibration.gguf.calibration/ directory records the corpus, per-request raw GGUF counts, routing JSONL, emitted token IDs, actual decode rows, stop reasons, hashes and REPORT.json. Prompt file contents are identified by hash; keep the original files to reproduce collection. The final profile appears only after all requests finish and the codec roundtrip succeeds. Use fresh output paths.

Bounds: 128 requests, 32 workload labels, an 8 MiB corpus file, 64 KiB/4096 tokens per prompt, 2..4096 requested output tokens and 64 MiB of workload-rate storage. Routing evidence is capped at 256 MiB per request; exceeding a bound aborts collection without publishing a final profile. Collection covers target decode logical accesses for raw and fused expert bodies, including all CPU/GPU ownership classes. Prefill, draft/MTP accesses and acceptance are excluded. The first token sampled from prefill has no decode row; the denominator counts actual subsequent target decode calls. Different hardware/placement can change greedy continuations. This method alone establishes neither held-out quality nor a speed gain for every model.

The older fixed-row diagnostic remains available through --test-moe-replay and GGML_TEST_MOE_PROFILE_EXPORT. It can continue past EOG and should not replace natural-EOG corpus calibration.

Load a profile and enable adaptation

./build/bin/llama-server -m model.gguf --moe-hybrid on \
  -ngl all -fa on -c 16384 -b 2048 -ub 2048 -np 1 -t 16 \
  --moe-expert-cache-size 64 --moe-expert-cache-host-pinned-mb 0 \
  --moe-expert-profile calibration.gguf --moe-profile-adapt occurrence \
  --load-mode none --lazy-mode on --fit off \
  --decode-overlap --decode-boundary-overlap --ple-prefetch --phase-aware-workspace

Load the model-bound statistics with --moe-expert-profile calibration.gguf. Default adaptation is off; add --moe-profile-adapt occurrence for the asynchronous occurrence policy or occurrence-sync for its synchronous control. Ranked STRP input remains supported when its geometry matches. Source names/domains, counts, types, shapes and complete bank coverage are checked before use. Binding does not fingerprint weight contents; regenerate or reevaluate profiles when model weights or quantization change. A separately loaded draft can use --spec-draft-moe-expert-profile and --spec-draft-moe-profile-adapt; target and draft configurations are independent.

The historical benchmark commands above remain pinned to 081cf1d79 and retain their recorded environments.

GenerelSchwerz llama.cpp

Feature groups

Source branches

Clone this wiki locally