Qwen3.8 Flash Next Reproduction

Qwen3.8 Flash Next reproduction protocol

This page reproduces the five owner-verified Qwen3.8 Flash Next runs listed in Notable runs. It fixes the source, models, build, prompts, request settings, launch commands, measurement rules, output hashes, and expected telemetry.

Important

Run each case from a fresh server started with --no-warmup. Send one measured request, stop the server, and confirm that its process, port, and GPU allocation clear. Reusing a server changes cache and page state.

Recorded machine

Component	Recorded value
Date	September 14, 2026
GPU	NVIDIA GeForce RTX 5070 Ti; 16,303 MiB reported total
CPU	Intel Core Ultra 9 285K; 24 cores / 24 logical CPUs
System RAM	65,160,504 KiB, or 62.14 GiB usable
Storage	ADATA LEGEND 800 GOLD NVMe; Btrfs
OS	CachyOS Linux x86_64
NVIDIA driver	`610.57.04`
CUDA Toolkit	`13.3.73`
Compiler	GCC `16.2.1 20260810`
Server threads	12

Different hardware can change speed and output even at temperature zero. Preserve the discrepancy instead of relabeling it as the original result.

The commands keep -t 12 and every other tuning value because this page reproduces recorded runs. For normal deployment, remove fixed CPU tuning unless you have measured it on your host. Retune thread and batch-thread counts, affinity, polling, priority, NUMA, context, batch, ubatch, parallelism, GPU placement, KV types, cache size, and host-pinned budget.

Get the exact source and models

git clone --branch moe-cache https://github.com/GenerelSchwerz/llama.cpp.git llama-qwen38-repro
cd llama-qwen38-repro
git checkout --detach 92593380102129b68a5c526d83cb59eb608a0b6d

hf download unsloth/Qwen3.8-Flash-Next-GGUF \
  --revision 38bb39ee97821de2c9009abb7e93950eec396e66 \
  --include 'UD-Q3_K_XL/*' \
  --include 'MTP/mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf' \
  --local-dir ./models/Qwen3.8-Flash-Next-GGUF

Verify every model file before running:

File	Bytes	SHA-256
`UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf`	10,946,624	`f2ef4328929d8b8c8930e2856eef52128dd4ce3425302f04bc3c657431cc4c49`
`UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL-00002-of-00003.gguf`	49,983,253,824	`7d230e7c9421d868b89eebaf23033af0ea1a4e046956df00fb156814fb62346e`
`UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL-00003-of-00003.gguf`	39,992,153,376	`21d4f90f9cd7b7c3a1582667c20cb22f7b03de895b88a23bb20aaeaa44f2c199`
`MTP/mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf`	2,786,204,800	`b646ef60eaae2a9ed849e75f15f399629ca22633555e99e809959e95f22a1575`

sha256sum \
  ./models/Qwen3.8-Flash-Next-GGUF/UD-Q3_K_XL/*.gguf \
  ./models/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf

The three base shards occupy about 83.81 GiB. --load-mode none --lazy-mode on allocates the ordinary model and expert source while leaving the model-marked PLE table lazy. It does not allocate all 83.81 GiB in RAM.

Build the measured configuration

cmake -S . -B build-notable-64k \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/usr/bin/cc \
  -DCMAKE_CXX_COMPILER=/usr/bin/c++ \
  -DCMAKE_CUDA_COMPILER=/opt/cuda/bin/nvcc \
  -DCMAKE_CUDA_ARCHITECTURES=120a-real \
  -DGGML_CUDA=ON \
  -DGGML_NATIVE=ON \
  -DGGML_CCACHE=ON \
  -DGGML_BACKEND_DL=OFF \
  -DGGML_BLAS=OFF \
  -DGGML_CUDA_GRAPHS=ON \
  -DGGML_CUDA_FA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=OFF \
  -DGGML_CUDA_FORCE_MMQ=OFF \
  -DGGML_CUDA_FORCE_CUBLAS=OFF \
  -DGGML_CUDA_NCCL=ON \
  -DGGML_CUDA_COMPRESSION_MODE=size

cmake --build build-notable-64k --target llama-server -j20

The recorded binaries had these hashes:

llama-server     7c815c4e48383b935356e6d33fb7f784b464a6a508d1996fcbd58b0becdeefd0
libggml-cuda.so  04d343c676e9a38771208933285164edb8634f35aec9eadef6769fce4bcb8399
libllama.so      c4235c320f0d293d28d7fef8d337642372a7e0700c47cc798b28902b0f48d466

A clean rebuild can have a different binary hash because paths and generated assets enter the build. The source commit, CMake cache, model hashes, server startup banner, and behavior are the primary identity checks.

Prepare the prompts

Copy the complete fixtures from Qwen3.8 Flash Next benchmark prompts into:

short.txt
long.txt

Then verify them:

test "$(wc -c < short.txt)" -eq 611
test "$(sha256sum short.txt | cut -d' ' -f1)" = 64759658f9c904e1fff03588f8a65b586f9f098c3b2e7d956adb1f56fe10cfeb

test "$(wc -c < long.txt)" -eq 27609
test "$(sha256sum long.txt | cut -d' ' -f1)" = 7b14ea48a1fbc13672693fb2606af5cfcca80de913c95f97580924c61dd5dcb4

The short request tokenizes to 158 prompt tokens. The long request tokenizes to 7,382 prompt tokens with this model and chat template.

Create an exact request

Save this as make-request.py:

import json
import pathlib
import sys

prompt_path, model, max_tokens, output_path = sys.argv[1:]
request = {
    "temperature": 0.0,
    "top_p": 1.0,
    "min_p": 0.0,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0,
    "seed": 12345,
    "cache_prompt": False,
    "stream": True,
    "model": model,
    "max_tokens": int(max_tokens),
    "stream_options": {"include_usage": True},
    "messages": [{"role": "user", "content": pathlib.Path(prompt_path).read_text()}],
    "ignore_eos": False,
    "return_tokens": True,
}
pathlib.Path(output_path).write_text(json.dumps(request, ensure_ascii=False, separators=(",", ":")))

This compact serialization reproduces each recorded request.json hash in the artifact manifest. Do not add a system message, request-side reasoning budget, UI wrapper, prompt-cache reuse, or sampling override. Run 6 sets its reasoning budget on the server command line.

Send and measure one request

Save this as measure.py. It preserves every SSE event and hashes reasoning and content separately.

import hashlib
import json
import pathlib
import sys
import time
import urllib.request

port, request_path, artifact_dir = sys.argv[1:]
artifact = pathlib.Path(artifact_dir)
artifact.mkdir(parents=True, exist_ok=False)
body = pathlib.Path(request_path).read_bytes()
(artifact / "request.json").write_bytes(body)
request = urllib.request.Request(
    f"http://127.0.0.1:{port}/v1/chat/completions",
    data=body,
    headers={"Content-Type": "application/json"},
)

events = []
content = []
reasoning = []
stamps = []
usage = None
finish_reason = None
server_timings = None
started = time.perf_counter()
with urllib.request.urlopen(request, timeout=360) as response:
    for raw_line in response:
        now = time.perf_counter()
        if not raw_line.startswith(b"data:"):
            continue
        payload = raw_line[5:].strip()
        if payload == b"[DONE]":
            break
        event = json.loads(payload)
        events.append(event)
        if event.get("usage"):
            usage = event["usage"]
        if event.get("timings"):
            server_timings = event["timings"]
        for choice in event.get("choices", []):
            delta = choice.get("delta", {})
            answer = delta.get("content") or ""
            thought = delta.get("reasoning_content") or ""
            content.append(answer)
            reasoning.append(thought)
            if answer or thought:
                stamps.append(now)
            if choice.get("finish_reason") is not None:
                finish_reason = choice["finish_reason"]
ended = time.perf_counter()

content_text = "".join(content)
reasoning_text = "".join(reasoning)
metrics = {
    "usage": usage,
    "finish_reason": finish_reason,
    "ttft_ms": (stamps[0] - started) * 1000,
    "decode_seconds": stamps[-1] - stamps[0],
    "decode_tok_s": (usage["completion_tokens"] - 1) / (stamps[-1] - stamps[0]),
    "request_wall_seconds": ended - started,
    "content_bytes": len(content_text.encode()),
    "content_sha256": hashlib.sha256(content_text.encode()).hexdigest(),
    "reasoning_bytes": len(reasoning_text.encode()),
    "reasoning_sha256": hashlib.sha256(reasoning_text.encode()).hexdigest(),
    "server_timings": server_timings,
}
(artifact / "events.json").write_text(json.dumps(events, ensure_ascii=False, indent=2) + "\n")
(artifact / "content.txt").write_text(content_text)
(artifact / "reasoning.txt").write_text(reasoning_text)
(artifact / "metrics.json").write_text(json.dumps(metrics, ensure_ascii=False, indent=2) + "\n")
print(json.dumps(metrics, indent=2))

The recorded client decode rate uses (completion_tokens - 1) / (last generated event - first generated event). Server eval time remains the primary direct comparison. Do not substitute curl time-to-first-byte for generated-text TTFT.

Clean launch environment

Before every run:

unset GGML_CUDA_MOE_FREQUENCY
unset GGML_CUDA_NO_PINNED
unset GGML_CUDA_MOE_EARLY_ROUTER_NATIVE
unset GGML_CUDA_MOE_EARLY_ROUTER_COPY_ENGINE
unset GGML_CUDA_MOE_EARLY_ROUTER_COPY_MAILBOX
unset GGML_CUDA_MOE_EARLY_ROUTER_COPY_POLL
unset GGML_CUDA_MOE_EARLY_ROUTER_COPY_BATCH
unset GGML_CUDA_MOE_EARLY_ROUTER_COPY_SPLIT
unset GGML_CUDA_MOE_EARLY_ROUTER_COPY_READY_ONLY
unset GGML_CUDA_MOE_EARLY_ROUTER_COPY_BANKS
unset GGML_CUDA_MOE_EARLY_ROUTER_COPY_DEBUG
unset GGML_CUDA_MOE_EARLY_ROUTER_STAGE_BLOCKS

export MODEL="$PWD/models/Qwen3.8-Flash-Next-GGUF/UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf"
export MTP="$PWD/models/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf"
export SERVER="$PWD/build-notable-64k/bin/llama-server"
export LD_LIBRARY_PATH="$PWD/build-notable-64k/bin"
export CUDA_VISIBLE_DEVICES=0
export GGML_CUDA_MOE_EARLY_ROUTER=1
export GGML_CUDA_MOE_EARLY_ROUTER_LOOKAHEAD=1
export SOURCE_COMMIT=92593380102129b68a5c526d83cb59eb608a0b6d
export COMPILE_OFF=1
export LLAMA_ATTN_ROT_DISABLE=1

Keep GGML_CUDA_MOE_FREQUENCY unset. The recorded runs used the default frequency-aware policy. These exact historical runs predate the CLI option and therefore retain the environment settings; use --moe-early-router for current builds.

Run each server command in terminal A. In terminal B, wait for the matching health endpoint before invoking make-request.py and measure.py:

curl --fail --silent --show-error http://127.0.0.1:PORT/health

Replace PORT with the port listed for that run. Do not send any other inference request to that server.

Run 1: 64K F16 full pin with MTP

Inputs

Item	Exact value
Prompt	`short.txt`; 611 bytes; SHA-256 `64759658f9c904e1fff03588f8a65b586f9f098c3b2e7d956adb1f56fe10cfeb`
Prompt / completion tokens	158 / 1,024
Alias	`moe-cache-q3xl-mtp-cache56-fullpin-f16-c64k`
Context / batch / ubatch	65,536 / 512 / 512
KV	F16 K / F16 V
Cache	56 slots
Host pinning	Full; host-budget flag omitted
Speculation	Integrated MTP, depth 2, Q4_K_M sidecar
Port / timeout	24626 / 360 seconds

Launch and request

set -o pipefail
timeout -k 5s 360s "$SERVER" \
  --offline --model "$MODEL" -md "$MTP" \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  -c 65536 -b 512 -ub 512 -np 1 -t 12 \
  -ngl all -fa on -fit off \
  --load-mode none --lazy-mode on \
  --moe-expert-cache-size 56 \
  -ctk f16 -ctv f16 -kvo \
  --cache-ram 0 --jinja --no-warmup \
  --backend-sampling --decode-overlap --decode-boundary-overlap \
  --ple-prefetch --phase-aware-workspace --live-context-workspace \
  --experimental-logs -n 1024 --log-colors off -lv 4 \
  --alias moe-cache-q3xl-mtp-cache56-fullpin-f16-c64k \
  --host 127.0.0.1 --port 24626 \
  2>&1 | tee run1-server.log

After /health returns {"status":"ok"}:

python3 make-request.py short.txt moe-cache-q3xl-mtp-cache56-fullpin-f16-c64k 1024 run1-request.json
python3 measure.py 24626 run1-request.json run1-response

Expected result

Check	Recorded value
Server prefill	136.83 tok/s; 1,154.69 ms
Server decode	61.68 tok/s; 16,585.02 ms
Client decode	61.681645 tok/s
TTFT / wall	1,162.979 ms / 17.751293 s
Loaded / peak VRAM	13,795 / 14,467 MiB
Finish	`length`; exact 158 + 1,024 = 1,182 tokens
Reasoning	4,481 bytes; SHA-256 `68f52c84f23ee79193469df079231e274ce14b6f56f132420460b49419946652`
Content	0 bytes; SHA-256 `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`
MTP	579 accepted / 887 drafted; 65.276%
Grouped	49 registered and covered; 22,200 completed calls; zero fallback, rollback, prepare, or finish errors
Cache total	24,135 hits; 16,202 misses; 8,138 evictions

The full expected response appears in Qwen3.8 Flash Next expected outputs.

Run 2: 64K Q8 full pin, short prompt

Inputs

Item	Exact value
Prompt	`short.txt`; 611 bytes; SHA-256 `64759658f9c904e1fff03588f8a65b586f9f098c3b2e7d956adb1f56fe10cfeb`
Prompt / completion tokens	158 / 1,024
Alias	`moe-cache-all-flags-fullpin-64k`
Context / batch / ubatch	65,536 / 4,096 / 512
KV	Q8_0 K / Q8_0 V
Cache	80 slots
Host pinning	Full; host-budget flag omitted
Speculation	Off
Port / timeout	24626 / 360 seconds

Launch and request

set -o pipefail
timeout -k 5s 360s "$SERVER" \
  --offline --model "$MODEL" --spec-type none \
  -c 65536 -b 4096 -ub 512 -np 1 -t 12 \
  -ngl all -fa on -fit off \
  --load-mode none --lazy-mode on \
  --moe-expert-cache-size 80 \
  -ctk q8_0 -ctv q8_0 -kvo \
  --cache-ram 0 --jinja --no-warmup \
  --backend-sampling --decode-overlap --decode-boundary-overlap \
  --ple-prefetch --phase-aware-workspace --live-context-workspace \
  --experimental-logs -n 1024 --log-colors off -lv 4 \
  --alias moe-cache-all-flags-fullpin-64k \
  --host 127.0.0.1 --port 24626 \
  2>&1 | tee run2-server.log

python3 make-request.py short.txt moe-cache-all-flags-fullpin-64k 1024 run2-request.json
python3 measure.py 24626 run2-request.json run2-response

Expected result

Check	Recorded value
Server prefill	115.86 tok/s; 1,363.73 ms
Server decode	55.54 tok/s; 18,419.85 ms
Client decode	55.540247 tok/s
TTFT / wall	1,389.713 ms / 19.813686 s
Loaded / peak VRAM	14,681 / 15,145 MiB
Finish	`length`; exact 158 + 1,024 = 1,182 tokens
Reasoning	4,596 bytes; SHA-256 `fcc014df69b36b44941cdfe6f74ba2c4661b9a48917a001bf9c404b79dba3e3f`
Content	0 bytes; SHA-256 `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`
Grouped	48 registered and covered; 49,104 completed calls; 1,018 graph reuses; zero fallback, rollback, prepare, or finish errors
Cache total	30,957 hits; 20,847 misses; 9,510 evictions
Decode overlap	1,023 queued; 0 discarded

The full expected response appears in Qwen3.8 Flash Next expected outputs.

Run 3: 64K Q8 full pin, long prompt

Inputs

Item	Exact value
Prompt	`long.txt`; 27,609 bytes; SHA-256 `7b14ea48a1fbc13672693fb2606af5cfcca80de913c95f97580924c61dd5dcb4`
Prompt / completion tokens	7,382 / 64
Alias	`moe-cache-all-flags-fullpin-64k-prefill`
Context / batch / ubatch	65,536 / 4,096 / 512
KV	Q8_0 K / Q8_0 V
Cache	80 slots
Host pinning	Full; host-budget flag omitted
Speculation	Off
Port / timeout	24627 / 360 seconds

Launch and request

set -o pipefail
timeout -k 5s 360s "$SERVER" \
  --offline --model "$MODEL" --spec-type none \
  -c 65536 -b 4096 -ub 512 -np 1 -t 12 \
  -ngl all -fa on -fit off \
  --load-mode none --lazy-mode on \
  --moe-expert-cache-size 80 \
  -ctk q8_0 -ctv q8_0 -kvo \
  --cache-ram 0 --jinja --no-warmup \
  --backend-sampling --decode-overlap --decode-boundary-overlap \
  --ple-prefetch --phase-aware-workspace --live-context-workspace \
  --experimental-logs -n 64 --log-colors off -lv 4 \
  --alias moe-cache-all-flags-fullpin-64k-prefill \
  --host 127.0.0.1 --port 24627 \
  2>&1 | tee run3-server.log

python3 make-request.py long.txt moe-cache-all-flags-fullpin-64k-prefill 64 run3-request.json
python3 measure.py 24627 run3-request.json run3-response

Expected result

Check	Recorded value
Server prefill	442.35 tok/s; 16,688.04 ms
Server decode	48.52 tok/s; 1,298.51 ms
Client decode	48.522591 tok/s
TTFT / wall	16,705.080 ms / 18.009455 s
Loaded / peak VRAM	14,681 / 15,571 MiB
Finish	`length`; exact 7,382 + 64 = 7,446 tokens
Reasoning	326 bytes; SHA-256 `a1e4332df0d126aeb2e514f88f95551c1452a32f09a338a3797a1797c846da7b`
Content	0 bytes; SHA-256 `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`
Grouped	48 registered and covered; 3,024 completed calls; zero fallback, rollback, prepare, or finish errors
Prefill cache	121,875 hits; 143,551 misses; 132,271 evictions
Prefill staging	0 staging tiles; 0 CPU-pack MiB; 0 upload errors

The full expected response appears in Qwen3.8 Flash Next expected outputs.

Run 4: 64K Q8 partial pin, short prompt

Inputs

Item	Exact value
Prompt	`short.txt`; 611 bytes; SHA-256 `64759658f9c904e1fff03588f8a65b586f9f098c3b2e7d956adb1f56fe10cfeb`
Prompt / completion tokens	158 / 1,024
Alias	`moe-cache-all-flags-pin24gb-c64k-decode`
Context / batch / ubatch	65,536 / 4,096 / 512
KV	Q8_0 K / Q8_0 V
Cache	80 slots
Host pinning	22,888 MiB model-wide budget
Speculation	Off
Port / timeout	24630 / 360 seconds

The 22,888 MiB setting equals 23,999,807,488 bytes, about 24 GB decimal. This Linux run exercises the Windows-shaped bounded path; it does not measure native WDDM.

Launch and request

set -o pipefail
timeout -k 5s 360s "$SERVER" \
  --offline --model "$MODEL" --spec-type none \
  -c 65536 -b 4096 -ub 512 -np 1 -t 12 \
  -ngl all -fa on -fit off \
  --load-mode none --lazy-mode on \
  --moe-expert-cache-size 80 \
  --moe-expert-cache-host-pinned-mb 22888 \
  -ctk q8_0 -ctv q8_0 -kvo \
  --cache-ram 0 --jinja --no-warmup \
  --backend-sampling --decode-overlap --decode-boundary-overlap \
  --ple-prefetch --phase-aware-workspace --live-context-workspace \
  --experimental-logs -n 1024 --log-colors off -lv 4 \
  --alias moe-cache-all-flags-pin24gb-c64k-decode \
  --host 127.0.0.1 --port 24630 \
  2>&1 | tee run4-server.log

python3 make-request.py short.txt moe-cache-all-flags-pin24gb-c64k-decode 1024 run4-request.json
python3 measure.py 24630 run4-request.json run4-response

Expected result

Check	Recorded value
Server prefill	31.49 tok/s; 5,016.94 ms
Server decode	36.18 tok/s; 28,271.96 ms
Client decode	36.192224 tok/s
TTFT / wall	5,033.518 ms / 33.302932 s
Loaded / peak VRAM	14,621 / 15,085 MiB
Finish	`length`; exact 158 + 1,024 = 1,182 tokens
Reasoning	4,596 bytes; SHA-256 `fcc014df69b36b44941cdfe6f74ba2c4661b9a48917a001bf9c404b79dba3e3f`
Content	0 bytes; SHA-256 `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`
Host admission	limit 23,999,807,488; direct source 23,370,141,696; staging reserve 218,746,880 bytes; 20 / 48 direct groups
Host peak	23,908,143,104 pinned bytes; 142,568,051,200 materialized bytes; 27,947 helper jobs
Grouped	48 registered and covered; 49,104 completed calls; zero fallback, rollback, prepare, or finish errors
Cache total	Same 30,957 hits, 20,847 misses, and 9,510 evictions as Run 2

The generated reasoning is byte-identical to Run 2. A different hash means the controlled full-pin/partial-pin parity no longer holds on that build or machine.

Run 5: 64K Q8 partial pin, long prompt

Inputs

Item	Exact value
Prompt	`long.txt`; 27,609 bytes; SHA-256 `7b14ea48a1fbc13672693fb2606af5cfcca80de913c95f97580924c61dd5dcb4`
Prompt / completion tokens	7,382 / 64
Alias	`moe-cache-all-flags-pin24gb-64k-prefill`
Context / batch / ubatch	65,536 / 4,096 / 512
KV	Q8_0 K / Q8_0 V
Cache	80 slots
Host pinning	22,888 MiB model-wide budget
Speculation	Off
Port / timeout	24628 / 360 seconds

Launch and request

set -o pipefail
timeout -k 5s 360s "$SERVER" \
  --offline --model "$MODEL" --spec-type none \
  -c 65536 -b 4096 -ub 512 -np 1 -t 12 \
  -ngl all -fa on -fit off \
  --load-mode none --lazy-mode on \
  --moe-expert-cache-size 80 \
  --moe-expert-cache-host-pinned-mb 22888 \
  -ctk q8_0 -ctv q8_0 -kvo \
  --cache-ram 0 --jinja --no-warmup \
  --backend-sampling --decode-overlap --decode-boundary-overlap \
  --ple-prefetch --phase-aware-workspace --live-context-workspace \
  --experimental-logs -n 64 --log-colors off -lv 4 \
  --alias moe-cache-all-flags-pin24gb-64k-prefill \
  --host 127.0.0.1 --port 24628 \
  2>&1 | tee run5-server.log

python3 make-request.py long.txt moe-cache-all-flags-pin24gb-64k-prefill 64 run5-request.json
python3 measure.py 24628 run5-request.json run5-response

Expected result

Check	Recorded value
Server prefill	194.62 tok/s; 37,930.30 ms
Server decode	33.69 tok/s; 1,870.01 ms
Client decode	33.690721 tok/s
TTFT / wall	38,009.643 ms / 39.893777 s
Loaded / peak VRAM	14,621 / 15,511 MiB
Finish	`length`; exact 7,382 + 64 = 7,446 tokens
Reasoning	326 bytes; SHA-256 `a1e4332df0d126aeb2e514f88f95551c1452a32f09a338a3797a1797c846da7b`
Content	0 bytes; SHA-256 `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`
Host admission	limit 23,999,807,488; direct source 23,370,141,696; staging reserve 218,746,880 bytes; 20 / 48 direct groups
Host peak	23,908,143,104 pinned bytes; 315,087,705,600 materialized bytes; 2,498 helper jobs
Grouped	48 registered and covered; 3,024 completed calls; zero fallback, rollback, prepare, or finish errors
Prefill cache	121,916 hits; 143,474 misses; 132,194 evictions
Prefill staging	406,986 staging tiles; 289,629.43 CPU-pack MiB; 20,991.508 CPU-pack ms; 0 upload errors

The generated reasoning is byte-identical to Run 3.

Run 6: earlier 12K Q8 public baseline

This historical run uses source commit b46f7f7a436f990932d3da3ec53380e2b9effc89, not the commit used by Runs 1 through 5. Build that revision in a separate checkout and never mix its libraries with the newer executable.

Source, build, and inputs

git clone https://github.com/GenerelSchwerz/llama.cpp.git llama-qwen38-47-repro
cd llama-qwen38-47-repro
git checkout --detach b46f7f7a436f990932d3da3ec53380e2b9effc89

hf download unsloth/Qwen3.8-Flash-Next-GGUF \
  --revision 38bb39ee97821de2c9009abb7e93950eec396e66 \
  --include 'UD-Q3_K_XL/*' \
  --local-dir ./models/Qwen3.8-Flash-Next-GGUF

cmake -S . -B build-ungated-cuda \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/usr/bin/cc \
  -DCMAKE_CXX_COMPILER=/usr/bin/c++ \
  -DCMAKE_CUDA_COMPILER=/opt/cuda/bin/nvcc \
  -DCMAKE_CUDA_ARCHITECTURES=120a-real \
  -DGGML_CUDA=ON \
  -DGGML_NATIVE=ON \
  -DGGML_CCACHE=ON \
  -DGGML_BACKEND_DL=OFF \
  -DGGML_BLAS=OFF \
  -DGGML_CUDA_GRAPHS=ON \
  -DGGML_CUDA_FA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=OFF \
  -DGGML_CUDA_FORCE_MMQ=OFF \
  -DGGML_CUDA_FORCE_CUBLAS=OFF \
  -DGGML_CUDA_NCCL=ON \
  -DGGML_CUDA_COMPRESSION_MODE=size

cmake --build build-ungated-cuda --target llama-server -j20

The recorded binaries had these hashes:

llama-server     2d0f27859186ae1f0e0ce846c5807f9dc9ce37a4a5bdfd4fc84f071065c9598b
libggml-cuda.so  ec7849d9096068f2657fce07b5b3d72fb8356cb01d7074e69a98c47dcef0f31b
libllama.so      7682fcf9aa0427aa119786ff2da04b1baa6c67bc8e8067d722f7fb47fbf3dea5

Item	Exact value
Prompt	`short.txt`; 611 bytes; SHA-256 `64759658f9c904e1fff03588f8a65b586f9f098c3b2e7d956adb1f56fe10cfeb`
Prompt / completion tokens	158 / 1,024
Alias	`early-router-flash`
Context / batch / ubatch	12,288 / 4,096 / 512
KV	Q8_0 K / Q8_0 V
Cache	80 slots
Host pinning	Full; this revision predates the bounded host-budget flag
Speculation / predictor	Off / absent from this revision
Reasoning budget	256 tokens, then `Now give the requested answer and finish all requested sections.`
Port / timeout	24625 / 240 seconds

Launch and request

export SERVER="$PWD/build-ungated-cuda/bin/llama-server"
export LD_LIBRARY_PATH="$PWD/build-ungated-cuda/bin"
export MODEL="$PWD/models/Qwen3.8-Flash-Next-GGUF/UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf"
export SOURCE_COMMIT=b46f7f7a436f990932d3da3ec53380e2b9effc89
export COMPILE_OFF=1
export LLAMA_ATTN_ROT_DISABLE=1
export CUDA_VISIBLE_DEVICES=0
unset GGML_CUDA_MOE_FREQUENCY
unset GGML_CUDA_MOE_EARLY_ROUTER
unset GGML_CUDA_MOE_EARLY_ROUTER_LOOKAHEAD

set -o pipefail
timeout -k 5s 240s "$SERVER" \
  --offline --model "$MODEL" \
  --reasoning-budget 256 \
  --reasoning-budget-message "Now give the requested answer and finish all requested sections." \
  --spec-type none \
  -c 12288 -b 4096 -ub 512 -np 1 -t 12 \
  -ngl all -fa on -fit off \
  --load-mode none --lazy-mode on \
  --moe-expert-cache-size 80 \
  -ctk q8_0 -ctv q8_0 -kvo \
  --cache-ram 0 --jinja --no-warmup \
  --experimental-logs -n 1024 --log-colors off -lv 3 \
  --alias early-router-flash \
  --host 127.0.0.1 --port 24625 \
  2>&1 | tee run6-server.log

python3 make-request.py short.txt early-router-flash 1024 run6-request.json
python3 measure.py 24625 run6-request.json run6-response

Expected result

Check	Recorded value
Server prefill	95.31 tok/s; 1,657.688 ms
Server decode	47.00 tok/s; 21,767.858 ms
Client decode	46.999212 tok/s
TTFT / wall	1,682.989 ms / 23.476999 s
Loaded / peak VRAM	13,908 / 14,384 MiB
Finish	`length`; exact 158 + 1,024 = 1,182 tokens
Reasoning	1,361 bytes; SHA-256 `18af1c0b8727d8c8790aefc38fcca3ee578ef12498db02b962d00d2f60a2e62e`
Content	2,922 bytes; SHA-256 `b6f509bedcdab739b071b42c666ca844b1acfacaec45172c2c3881c3c9cd13d1`
Grouped	48 registered and covered; 49,104 completed calls; zero fallback, rollback, prepare, or finish errors

Flash Next 47 tok/s evidence contains the full reasoning and answer streams and the complete terminal measurement record.

Diagnose a mismatch

Check these layers in order:

Input identity: source commit, model hashes, prompt bytes/hash, request fields, chat endpoint, alias, token counts, and fresh-server status.
Build identity: CUDA architecture, FA quant list, dynamic-backend choice, compiler, startup build banner, and matched libraries.
Path activation: CUDA_MoE_Cached model buffer size = 53237.50 MiB, nonzero cache pools, staged-input messages, and the expected direct-group count.
Correctness: exact token accounting, finish reason, coherent text, reasoning/content hashes, and the first differing returned token ID when available.
Cache execution: registered/covered groups, completed grouped calls, hits, misses, evictions, graph reuse, and all fallback/error counters.
Host movement: direct source bytes, staging tiles, CPU-pack MiB/ms, upload bytes, helper jobs, and pinned peak.
Performance: server prefill and eval timing first; then client TTFT/window, GPU clocks, PCIe state, CPU load, storage faults, RAM pressure, and profiling overhead.

Do not reject a future build only because its generated hash changed. Temperature zero does not guarantee cross-build or cross-GPU identity. First locate the earliest differing token and compare the execution counters. Reject the run as a reproduction only when an uncontrolled input, path regression, runtime error, malformed output, accounting failure, or material unexplained timing change exists.

Artifact manifest from the recorded runs

These hashes identify the preserved local artifacts. They do not replace the semantic checks above.

Local archive directories:

Run 1  /home/gencoolpc/moe-cache-tests/results/moe-cache-925933801-q3xl-mtp-cache56-c64k-fullpin-f16-20260914
Run 2  /home/gencoolpc/moe-cache-tests/results/moe-cache-925933801-all-flags-fullpin-c64k-q8-short-no-rbudget-20260914
Run 3  /home/gencoolpc/moe-cache-tests/results/moe-cache-925933801-all-flags-fullpin-c64k-q8-prefill7382-gen64-20260914
Run 4  /home/gencoolpc/moe-cache-tests/results/moe-cache-925933801-all-flags-pin24gb-c65536-q8-short158-gen1024-20260914
Run 5  /home/gencoolpc/moe-cache-tests/results/moe-cache-925933801-all-flags-pin24gb-c64k-q8-prefill7382-gen64-20260914
Run 6  /home/gencoolpc/moe-cache-tests/results/flash-c80-public-moe-cache-b46f7f7a4-chat1024-20260907

Run	`request.json`	`response.json`	`server.log`	`summary.txt`
Run 1	`2c18247f302a31612afbf3c996728c7eba9c911e3d9a28dc22b82fdcaa7ac83f`	`62c0971a0ea6ae2b83eeadcd7a666573f8b39f63aae92bf3ce108b9f22f64e27`	`2041f2386e2fd624251803d36a9914d6c33c1fddd9e569b8a102b58646163b87`	`1f12cb723f6805f3c7c8ee6d80e06b571b1a2fc3dcefa0166b1721b77c42b242`
Run 2	`b56a76dde0d1cf498b9c5cb7e283541197d68e713597b6d04e3a25a9a2ef8574`	`405eb98cc0f0b882b8f96b78989ab6258d8b90a63e3a7334e85f09504fc76010`	`12b608e9d874b01a2a7565bf358cafc02480c58fe889419353c428656a61de9b`	`3b7cee7a03e0e288d6262037d3dd2d8c301a115f66916d6b76891c26887ca096`
Run 3	`0da899f05ac0cf8255c71bbf95e903fab1063579d314ce65953478d97e02a643`	`e1027fedc63f4cb34c9f806c0b15e35b7be79568de1107c52f37bfe6ff044087`	`a0e98572d776ee9890d5f9bb43ad478a1fcba1af6ea6081a8b3199abe57fd9d4`	`3c9716f74008b7efa095f8041d225ec8c186af7fcad8e9758f4e84a90b48fa59`
Run 4	`d4f90415106a67203dac034aac7692e9917456c966790fa8f41d67d8502bb209`	`746910ee0cf2b022493248611160d3d2be7e73215c2accb81f882ce01547b969`	`ae28d4ec3c9e43fa84b572bb5f624b361be6368d90a8e3d3f81df60e707fd6a3`	`2994796f6d1f7b54655a334e930f8551343126949c46e5ea32ab6008699a2968`
Run 5	`5e45b3ef4d81a67acee8513937a369ebbc911f8cc881d3c91dcf5aa3b28f8c64`	`54e7288aba1309b996df32b0fabe813e7d37048e773fdf2346794bb2aa081004`	`31e57702d5011eff2021b2b9fa32ab1ec29bc43b459c1cba361dab98bc618742`	`43210d8e3f5d64ebb685528794b92354e530cdff0827961e6b23440dad7306ab`
Run 6	`6af2608e41ace1e3bb95a56f4fc660c818e058ae0a99761dffe841c6a5108be0`	`c4beca73b99ee3a0acab8b06f99d954389345c923a0c0a988ed0a1c6cb1541c2`	`7cd6aa3fdbc74c974b10b03976add5db25b1643e357ac119713cca6d26fa5164`	`44129943e1fc67cfff46c8e5911855695fc8be1e81b16cf023a9fd9034c90f45`

Exact prompts - Expected outputs - Notable runs - MoE diagnostics

GenerelSchwerz llama.cpp

Feature groups

Source branches

Qwen3.8 Flash Next Reproduction

Qwen3.8 Flash Next reproduction protocol

Recorded machine

Get the exact source and models

Build the measured configuration

Prepare the prompts

Create an exact request

Send and measure one request

Clean launch environment

Run 1: 64K F16 full pin with MTP

Inputs

Launch and request

Expected result

Run 2: 64K Q8 full pin, short prompt

Inputs

Launch and request

Expected result

Run 3: 64K Q8 full pin, long prompt

Inputs

Launch and request

Expected result

Run 4: 64K Q8 partial pin, short prompt

Inputs

Launch and request

Expected result

Run 5: 64K Q8 partial pin, long prompt

Inputs

Launch and request

Expected result

Run 6: earlier 12K Q8 public baseline

Source, build, and inputs

Launch and request

Expected result

Diagnose a mismatch

Artifact manifest from the recorded runs

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!