🐣 GLM-5.3-Flash (320B) and Qwen3.8-Flash-Next land on the club-3090 moe-cache engine

[noonghunna]

We just shipped nine slugs on the llamacpp-club3090 moe-cache engine, now at v1.6:

• llamacpp-club3090/glm53-flash-{dual,multi4,multi8}-{iq3xxs,iq4xs}-moecache — GLM-5.3-Flash, a 320.76B MoE (arch glm5next) from Z.ai
• llamacpp-club3090/qwen38-flash-next-{dual,multi4,multi8}-q4kxl-moecache — Qwen3.8-Flash-Next (arch qwen4exp)

Both run on two 24 GB consumer cards by keeping the routed experts in host RAM and caching hot experts back onto the GPUs. GGUF quants by @unsloth. The glm5next and qwen4exp architectures are vendored from @danielhanchen's upstream PRs; the expert cache is leloch's work (RFC ggml-org#24528). We maintain neither — what we contribute is measurement.

---

The new slugs

All prefixed llamacpp-club3090/. Context is 204,800 on every one.

Slug	Cards	Quant	Host RAM	Port
glm53-flash-dual-iq3xxs-moecache	2	unsloth UD-IQ3_XXS	~114 GB	8125
glm53-flash-multi4-iq3xxs-moecache	4	″	~114 GB	8126
glm53-flash-multi8-iq3xxs-moecache	8	″	~114 GB	8127
glm53-flash-dual-iq4xs-moecache	2	unsloth UD-IQ4_XS	~147 GB	8119
glm53-flash-multi4-iq4xs-moecache	4	″	~147 GB	8120
glm53-flash-multi8-iq4xs-moecache	8	″	~147 GB	8121
qwen38-flash-next-dual-q4kxl-moecache	2	unsloth UD-Q4_K_XL	~110 GB	8122
qwen38-flash-next-multi4-q4kxl-moecache	4	″	~110 GB	8123
qwen38-flash-next-multi8-q4kxl-moecache	8	″	~110 GB	8124

All 🐣 Incubating — --force required, and hidden from switch.sh --list (use --list --all).

Only the two dual slugs have booted here. The 4- and 8-card variants are community-validated by
design: this rig has exactly two 3090s, permanently, so the first person to boot one is the
validation.

🐣 Why Incubating, and how to read the numbers below

Same reasoning as the DeepSeek-V4-Flash slugs (#909), and it has not changed: these are not GPU-bound. Under expert offload the GPU waits on host memory, so what you get is set by your RAM channels, their speed, and your core count — not by having two 3090s. A decode number from this rig would not predict yours.

We publish this rig's absolute numbers below anyway — a datapoint beats silence — but read them
as what this machine did, never as a forecast for yours. The DeepSeek-V4-Flash slugs shipped
with none at all; we think that was too cautious.

Two results here do travel, because both are same-rig A/Bs: the v1.6 engine gain, and the
micro-batch trade further down. Ratios survive a change of hardware in a way absolutes don't.

The v1.6 engine — +46.9% decode at 44K, +69.7% at 87K

v1.6 carries two commits from @danielhanchen's #27754: 0069971 (caches the indexer's pooled keys) and 5796547f3 (invalidates them when a context shift splits a pool). Measured with and without the two commits — same rig, same weights, 4 boots alternating:

KV depth	without	with	Δ
~11	15.45	16.14	+4.5%
10,932	13.75	15.71	+14.3%
43,680	10.15	14.91	+46.9%
87,343	7.98	13.55	+69.7%

Decode falloff from shallow to 87K goes from −48% to −16%. It is a ratio on one machine, so it should survive the trip to yours even though the absolute numbers won't.

⚠️ The two commits are inseparable, and this is worth knowing if you build your own. With 0069971 alone, generation at depth truncates — 60 tokens where the baseline emitted 128, on the same prompt at temperature 0, reproducible across boots. It is not a crash and not a slowdown: pooled keys go stale while still looking complete, so the sparse attention selects the wrong cells. 5796547f3 removes it. Never take one without the other.

Why this is a decode-at-depth story at all

The effect is invisible at short context. Our own first A/B said "no reproducible benefit" — because our bench measured decode only at shallow context and measured the deep legs as prefill. At ~11 tokens of KV the gain really is +4.5%, which was all we were looking at. We added a decode-at-depth leg to the bench and the +70% appeared. If you are evaluating a cache-shaped change, check that your harness exercises the regime it targets.

The numbers — Qwen3.8-Flash-Next, 2× RTX 3090

Canonical bench (3 warm + 5 measured), on the shipped compose as it boots — moe-cache v1.6,
q8_0 KV, 204,800 context, no draft model (this GGUF carries no MTP head) but with
ngram-mod speculation ON at n=8, which is the shipped default and needs no drafter file:

	value	CV
decode, narrative	36.83 tok/s	1.3%
decode, code	36.52 tok/s	2.5%
prefill @10k	787 tok/s	0.6%
TTFT (short prompt)	~165 ms	—
moe-cache hit rate	67.1% / 61.4% per card	—

verify-full 10/10, all checks passed, including vision on ground truth.

⚠️ Read these as "what this rig did", not "what your rig will do". This is a CPU-offload
config: the experts stream from host RAM, so your number is set by your memory channels, their
speed, and your core count — the same reason the DeepSeek-V4-Flash slugs (#909) shipped with no
throughput figures at all. We publish these because a 36 tok/s datapoint is more useful than
silence, not because they will transfer. A rig with half the memory bandwidth will not see 36.

The numbers — GLM-5.3-Flash, 2× RTX 3090

Canonical bench (3 warm + 5 measured) on the shipped dual compose as it boots — moe-cache v1.6,
IQ3_XXS, fp16 KV, 204,800 context, DFlash2 drafter on GPU1 at depth 3, thinking on
(REASONING_EFFORT=max, GLM's vendor default):

	value	CV
decode, narrative	18.12 tok/s	3.9%
decode, code	18.47 tok/s	3.2%
prefill @10k	340 tok/s	1.2%
decode @10k depth	13.10 tok/s	11.4%
moe-cache hit rate	46.0% / 45.5% per card	—

⚠️ Thinking-inclusive. GLM thinks by default and the bench counts reasoning_content, so the
user-visible answer lands later than the raw tok/s implies. The same caveat as Qwen applies twice
over here: this is a CPU-offload config and a 320B model — your memory bandwidth sets your number.

verify-full 9/10, not a clean sweep. The failure is check 7 (thinking/reasoning): the model
reasoned "Simple." — 7 characters — against a harness minimum. It failed identically on both arms
of the ubatch A/B below, so it is not a regression, but whether it is GLM being terse or a threshold
mis-set for this model is untriaged. Vision passed 4/4 on ground truth.

⭐ The second number that travels: halving the micro-batch buys ~24% decode

Same rig, same image, same compose, only -ub/-b differing — so like the v1.6 result above, the
ratio should survive the trip to your hardware even though the absolutes will not:

	-ub 4096	-ub 2048	Δ
decode, narrative	14.58	18.12	+24.3%
decode, code	14.93	18.47	+23.7%
prefill @10k	415	340	−18.2%

Decode ranges do not overlap on either leg. The mechanism is specific to glm5next: its kpool
sparse-indexer masks are ~44-45% of the compute buffer and scale with n_ubatch, so halving it
hands that VRAM back to the expert cache. It does not carry to Qwen3.8-Flash-Next — qwen4exp
has no kpool, and we have not measured it there, so those composes are untouched at 4096.

Break-even is prompt ≈ 25× response tokens: below that, 2048 wins; above it, 4096 does. A
forced-thinking model emits long responses, so ordinary interactive turns sit well inside the 2048
region — which is why the dual composes now default to it. multi4/multi8 keep 4096, where
long-context prefill dominates. For bulk long-document prefill on dual, set UBATCH=4096 in your
environment before launching.

⚠️ GLM thinks, and you cannot turn it off per request

Worth knowing before you wire GLM into anything, because it surprised us: no per-request thinking
switch works on this build. We tried four request shapes — reasoning_effort in the body,
chat_template_kwargs.enable_thinking, /nothink in the prompt, and the OpenAI-style
reasoning: {effort} object — and all four were inert. Thinking is controlled server-side only,
by the REASONING_EFFORT environment variable on the compose, which defaults to max (the vendor
default).

So: to run GLM with thinking effectively off, relaunch with REASONING_EFFORT=low — you cannot do
it from the client. Every throughput figure above is thinking-inclusive as a result, and an agent
loop that assumes it can disable reasoning for a cheap turn will not get one.

What you actually need to run these

Host RAM is the hard gate. The experts live in host memory, so this is not negotiable:

model / quant	host RAM	disk (weights + companions)	GPUs	context
GLM-5.3-Flash IQ3_XXS	~114 GB	113 GB + 3 GB	2× 24 GB	204,800
GLM-5.3-Flash IQ4_XS	~147 GB	147 GB + 3 GB	2× 24 GB	204,800
Qwen3.8-Flash-Next UD-Q4_K_XL	~110 GB	104 GB + 0.9 GB	2× 24 GB	204,800

Below the RAM figure the box thrashes or OOMs; preflight refuses to launch and prints the number.
The companions are the DFlash2 drafter (1.9 GB, GLM only) and the vision projector (~1 GB). Host RAM
is set by the model, not the topology — the multi4/multi8 variants need the same amount.

CPU: core count is a first-class variable here, not a footnote. These composes default -t to
nproc/2, which is portable but is a floor, not the optimum — on our 32-core host -t 28
measured ~20% faster than nproc/2=16. The useful zone is an absolute ~24-32 threads, not a
fraction of your core count, so it is worth setting explicitly for your box:

THREADS=<your core count, roughly> bash scripts/switch.sh --force <slug>

The useful zone we measured is an absolute ~24-32 threads, not a fraction of your core count.

The reference rig — so you can calibrate the numbers above against your own box:

	this rig
GPUs	2× RTX 3090 (24 GB), PCIe, no NVLink
CPU	AMD EPYC 7543, 32 cores
Memory	196 GB, 8 channels

⚠️ That memory-channel count is the single biggest reason our throughput will not transfer. An
8-channel server platform feeds CPU-resident experts far faster than a 2-channel desktop, at the
same clock. If you are on a mainstream desktop board, expect materially lower decode than the tables
above — that is the platform, not the config, and it is why we keep saying the ratios travel and
the absolutes don't.

What the two models give you

GLM-5.3-Flash ships in two quant tiers — IQ3_XXS (the reach tier, 33 GB less host RAM) and IQ4_XS. Both carry vision (glm5next CLIP projector) and both ship the external DFlash2 drafter pinned to GPU1 at depth 3.

Qwen3.8-Flash-Next carries vision, and runs q8_0 KV rather than fp16 — which measured
−47% KV footprint and converts directly into expert-cache slots.

It ships n-gram speculation on by default (ngram-mod, n=8) — that needs no drafter file,
being a runtime spec-type rather than a -md path. On this model it measured +64% on repeated
context and −1.3% (noise) on first exposure, for ~34 MiB. The gain is cache-residency dependent,
so it pays on multi-turn work over one file and costs nothing measurable on cold traffic. ⛔ Don't
raise the depth: ngram-mod's own default (n_max=64) measured −7.6% here, because a wide verify
batch unions the expert set and every non-resident expert becomes a CPU-side FFN.

What it does not have is a draft model, and the reason is worth stating precisely because
it is an engine gap, not a weights gap:

• The unsloth GGUF we serve carries no MTP head (nextn_predict_layers absent).
• An MTP head does exist — @dzannotti's Qwen3.8-Flash-Next-MTP-GGUF publishes a standalone qwen4exp drafter (blk.48.nextn.*, nextn_predict_layers = 1) and a re-shard of the very UD-Q4_K_XL quant we run, with the head added.
• Our engine cannot use it yet. qwen4exp has no LLM_GRAPH_TYPE_DECODER_MTP implementation — 15 other architectures in this build do, including glm5next, which only gained one upstream on 2026-08-30. Until then the head would load as unused tensors.
• The fix is in flight upstream: ggml-org/llama.cpp#27836 adds exactly that draft head for qwen4exp. When it lands, this slug gets a speculative path.

So Qwen Flash runs drafter-free today, and that is a re-test trigger with a PR number attached, not a permanent property.

⭐ And more has landed since — though not in this image. #27977 — "reduce the generation slowdown as context grows", the qwen4exp analogue of the very change that bought GLM +69.7% at 87K depth in this release — was closed on 2026-09-01, not rejected: its author split it into one PR per change. Three of that series are now merged upstream — #28011 (kv-cells: stop the sequence scan once all sequences are seen, merged 2026-08-30), #28023 (sum the indexer heads by slices) and #28123 (recurrent state rollback), both merged 2026-09-01.

⚠️ None of them are in the v1.6 image benchmarked here. v1.6 was built 2026-08-31 with no rebase — its base is still v1.5's b10675, carrying only the two cherry-picked #27754 commits — and two of the three merged after that build in any case. So Qwen Flash's 36.8 tok/s stands exactly as measured, and the upside is a re-test trigger with merged PR numbers behind it rather than an open one, pending an engine rebase.

What is NOT tested

Stated plainly, because 🐣 means what it says:

• No quality numbers. No 8-pack on either model, at any quant. IQ3_XXS in particular is a 3-bit expert path.
• No soak and no verify-stress on the v1.6 binary. verify-full was run on both models (Qwen 10/10; GLM 9/10 — see above), along with test-moe-cache (44 cases, identical across both prefetch states), a retrieval-correctness gate at 8K/32K/64K, and each dual slug booted on the published image showing expert offload, non-zero cache hits and correct terminating output.
• The ubatch result is decode-and-prefill only. Pool residency was not compared between the two arms (the 2048 arm ran without the pool parser), so the reclaim mechanism above is inferred from earlier measurements on an older engine build rather than re-measured here. Prefill was measured at 10K only — in those earlier measurements the penalty shrank with depth, which would make the trade better at long context than the 10K figure suggests, but that is untested here. Output was not quality-checked at the new ubatch.
• multi4 / multi8 are community-validated by design. This rig has exactly two 3090s. The first person to boot one is the validation.
• The GLM drafter's own contribution is unquantified on this engine. DFlash2 ships on by default (GPU1, depth 3) and every GLM figure above was measured with it active, so those numbers stand as published. What we have not re-measured is how much of the throughput is the drafter: its margin narrowed on the new engine, because the indexer-cache commits raised the no-drafter baseline while the drafter itself did not change. Its depth (n=3) was likewise tuned against a slower target and may no longer be optimal. If you A/B it with SPEC_N=0, that is a result we would like to see.

Getting it

setup.sh pulls the weights and the required drafter; the vision projector is opt-in.

GLM-5.3-Flash — reach tier (IQ3_XXS, ~114 GB host RAM)

WITH_VISION=1 WEIGHT_KEY=glm-5.3-flash:unsloth-ud-iq3xxs bash scripts/setup.sh glm-5.3-flash

⚠️ Name the quant explicitly. The model's default is IQ4_XS, so a bare setup.sh glm-5.3-flash
downloads the larger tier and the iq3xxs slug then finds no weights.

GLM-5.3-Flash — default tier (IQ4_XS, ~147 GB host RAM)

WITH_VISION=1 bash scripts/setup.sh glm-5.3-flash

Qwen3.8-Flash-Next (~110 GB host RAM)

WITH_VISION=1 bash scripts/setup.sh qwen3.8-flash-next

	GLM	Qwen
weights	✅ the quant you named	✅ UD-Q4_K_XL (only one)
DFlash2 drafter	✅ always — not optional	— none exists for qwen4exp
mmproj (vision)	only with WITH_VISION=1	only with WITH_VISION=1

⚠️ Omitting WITH_VISION=1 does not break the boot — the compose detects the missing projector,
prints the fix, and serves text-only. A silent capability loss, not a crash.

Run it

GLM-5.3-Flash

bash scripts/switch.sh --force llamacpp-club3090/glm53-flash-dual-iq3xxs-moecache   # ~18 tok/s · :8125
bash scripts/switch.sh --force llamacpp-club3090/glm53-flash-dual-iq4xs-moecache    # :8119

Qwen3.8-Flash-Next

bash scripts/switch.sh --force llamacpp-club3090/qwen38-flash-next-dual-q4kxl-moecache   # ~37 tok/s · :8122

Useful env, both models: THREADS=<n> (default is nproc/2 — portable, but a floor; the useful
zone is an absolute ~24-32) · SPEC_N=0 (speculation off) · UBATCH=4096 (GLM dual: trade
decode back for prefill on long documents).

GLM only: REASONING_EFFORT=low — the only way to reduce thinking, since no per-request
switch works. Qwen only: THINKING=1 — selects the thinking sampler row and --reasoning on.

Run the evals yourself

Two non-overlapping passes: behavioural quality (the 8-pack) and operational health. Both models
here are llama.cpp on the moe-cache engine, so there is no engine branching.

One-time setup — three of the eight packs run in Docker sandboxes that build once:

pip install git+https://github.com/noonghunna/benchlocal-cli.git
git clone https://github.com/noonghunna/benchlocal-cli
bash benchlocal-cli/tools/build-sandboxes.sh     # ~30 GB free; prune if tight

Without the images the 5 deterministic packs still run; the 3 sandboxed ones skip with a warning.

⚠️ Read this before you set a timeout — it is arithmetic, not taste

benchlocal's sandbox cap BENCHLOCAL_MODEL_TURN_TIMEOUT is 300 s flat and does not scale with
throughput. One 16,384-token thinking answer takes 16384 / decode_TPS seconds, so anything
slower than ~55 tok/s busts the default. Both models here are well under that:

model	decode	16,384 tokens takes	so set at least
GLM-5.3-Flash	~18 tok/s	~905 s	1200
Qwen3.8-Flash-Next	~37 tok/s	~445 s	900

Leave it at 300 and cli-40, hermesagent-20 and bugfind-15 fail as timeouts that read like
wrong answers. This is the single most likely way to get a bogus quality result on these slugs.

Quality — GLM-5.3-Flash

⚠️ GLM has no per-request thinking switch (four request shapes measured inert). Each leg needs
its own boot with REASONING_EFFORT, and the shipped default is max.

# leg A — thinking (the shipped default)
bash scripts/switch.sh --force llamacpp-club3090/glm53-flash-dual-iq3xxs-moecache
BENCHLOCAL_MODEL_TURN_TIMEOUT=1200 bash scripts/quality-test.sh --full --enable-thinking --sampling-from-server   --max-tokens 4096 --thinking-max-tokens 16384 --timeout-per-case 1200

# leg B — thinking effectively off  (server-side env; there is no client-side way)
REASONING_EFFORT=low bash scripts/switch.sh --force llamacpp-club3090/glm53-flash-dual-iq3xxs-moecache
BENCHLOCAL_MODEL_TURN_TIMEOUT=1200 bash scripts/quality-test.sh --full --no-thinking --sampling-from-server   --max-tokens 4096 --timeout-per-case 900

Quality — Qwen3.8-Flash-Next

Instruct is the shipped default, so leg A is the bare launch. THINKING=1 selects both the
thinking sampler row and --reasoning on, so unlike the vLLM composes you do not hand-set the
sampler.

# leg A — instruct (default; no env)
bash scripts/switch.sh --force llamacpp-club3090/qwen38-flash-next-dual-q4kxl-moecache
BENCHLOCAL_MODEL_TURN_TIMEOUT=900 bash scripts/quality-test.sh --full --no-thinking --sampling-from-server   --max-tokens 4096 --timeout-per-case 900

# leg B — thinking
THINKING=1 bash scripts/switch.sh --force llamacpp-club3090/qwen38-flash-next-dual-q4kxl-moecache
BENCHLOCAL_MODEL_TURN_TIMEOUT=900 bash scripts/quality-test.sh --full --enable-thinking --sampling-from-server   --max-tokens 4096 --thinking-max-tokens 16384 --timeout-per-case 900

⚠️ Each leg needs its own boot, in both directions. The container persists between runs, so
going back to instruct after a thinking run needs a reboot too — skip it and the "instruct" leg is
silently a second thinking leg, both arms score alike, and the A/B reads as a clean null. benchlocal
catches this: per-pack thinking_validity in the saved JSON.

Why each flag

Each of these exists because its absence produced a wrong number for us:

Flag	Why
--sampling-from-server	Uses the compose's card-correct sampler instead of the pack's temperature=0. Without it you measure greedy decoding, not the shipped config. ⚠️ Both models' bench numbers use the stack protocol sampler for cross-engine comparability — correct for TPS, wrong for quality.
--max-tokens 4096	The ~1024 default silently truncates long answers into token_limit failures that look like wrong answers.
--thinking-max-tokens 16384	The thinking arm needs the headroom — and on these models drives the timeout arithmetic above.
--timeout-per-case	Scales off a startup decode-TPS probe, but give it room: at 18 tok/s a long answer is minutes of wall clock.
BENCHLOCAL_MODEL_TURN_TIMEOUT	The flat sandbox cap. See the table above — this is the one that bites here.
--repeat 3	Add it for anything you'll quote — with --sampling-from-server both legs are sampled, so single draws aren't quotable.

⚠️ Read the failure modes before reading the score. timeout and token_limit are harness
artifacts; only verifier_fail / wrong_answer are the model. benchlocal retries model verdicts 3×
by default but does not retry timeouts, so a timeout row is a single sample against a clock.

Operational health

bash scripts/report.sh --full     # verify + stress + soak + bench + agentic, redacted

Don't pair that with rebench-full.sh — they re-run the same gates.

⚠️ Budget generously. These are CPU-offload configs at 18–37 tok/s, so every wall-clock default in
the suite is tighter here than on a GPU-resident model.

What would help most

Numbers from a rig that isn't this one. Host RAM channels and core count are the variables we cannot cover, and the multi4/multi8 variants have never been booted by anyone. If you run one, the numbers-from-your-rig issue template is the place.

Credits: GLM-5.3-Flash by Z.ai; Qwen3.8-Flash-Next by the Qwen team; GGUF quants and the DFlash2 drafter by @unsloth; the glm5next/qwen4exp architectures and both v1.6 commits by @danielhanchen (ggml-org/llama.cpp#27754); the MoE expert cache by leloch (RFC ggml-org#24528); DFlash2 runtime upstream by Xuan-Son Nguyen (#27342).

[gstephan]

Great work! I'm happily using DS4 Flash 0731 on llama.cpp + moecache ( a forked i made with a cache re-arm system that is missing from the original work), getting 20t/s with the biggest Q4 quant, it could be here too :)

[noonghunna]

we do have a slug for deepseek flash on moecache too.

[Whamp]

You may want to check this out for a different a vLLM based 2x3090 slug to incubate:

https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE
https://github.com/DominikBucko/qwen38-flash-next-2x3090

pretty impressive numbers on the tin:

Measured performance
On 2× RTX 3090 with 128 GB of system memory:

262,016-token prompt: 1,275.6 prompt token/s;
128-token decode after that prompt: 54.5 token/s;
warmed 128-input/4,096-output greedy probes: 127.1–134.0 output token/s;
MTP acceptance on the warm probes: 86.3–90.8%.

[shaxs]

I saw some pretty impressive numbers with this setup especially with 4 cards.

Matched two-GPU vs four-GPU results

Both long-context streams used one fresh candidate boot, a unique prompt prefix, a forced visible 128-token response with thinking disabled, and direct SSE timing. The prompt counts were effectively identical: 258,120 tokens on TP2 and 258,126 tokens on TP4.

Metric	TP2 result	TP4 result	Change
Warmed 128-in / 4,096-out decode, median of 5	119.79 tok/s	162.71 tok/s	+35.83%
4K decode range	n/a in this comparison record	158.75–163.20 tok/s	4.45 tok/s spread
Uncached ~258K prompt prefill	1,402.31 prompt tok/s	1,666.23 prompt tok/s	+18.82%
Time to first visible token after ~258K prompt	184.068 s	154.916 s	15.84% lower
Decode after ~258K context, 128 forced visible tokens	129.12 tok/s	135.80 tok/s	+5.17%
Prefix-cache hits before/after TP4 long stream	0 / 0	0 / 0	valid uncached measurement

TP4 4K decode detail

All five measured outputs were exactly 4,096 tokens, had finish_reason=length, and had reasoning disabled:

158.75, 161.66, 163.20, 162.78, 162.71 tok/s
median = 162.71 tok/s

TP4 long-context detail

Prompt tokens:             258,126
Completion tokens:         128
TTFT / prefill interval:   154.916 s
Prefill rate by TTFT:      1,666.23 tok/s
Visible decode interval:   0.94259 s
Visible decode rate:       135.80 tok/s
End-to-end request rate:   0.8213 completion tok/s
Finish reason:             length
Reasoning tokens:          0
Prefix-cache hits:         0 before, 0 after

The end-to-end completion rate is intentionally not a decode metric: it includes the full 154.9-second prefill.

Metrics after the TP4 long stream recorded 88 accepted speculative tokens. This is not reported as a full acceptance percentage because the matching draft-total counter was not captured in the same minimal metric snapshot.

Concurrency ladder — interpretation limited by configuration

The candidate completed all forced 512-token API requests with reasoning disabled. Aggregate rates were:

Simultaneous submitted requests	Aggregate median completion rate	Two measured batches
1	78.74 tok/s	78.48, 78.99
2	79.58 tok/s	78.35, 80.82
4	82.91 tok/s	84.01, 81.82

This is not a multi-request scaling verdict. The actual loaded TP4 engine log says max_num_seqs=1, so the server serializes/queues this workload by design. The ladder confirms stability under queued submissions; it does not show the capacity of a tuned multi-sequence TP4 profile.

[noonghunna]

Thank you! @shaxs could you please submit rig reports for both TP=2 and TP=4?

run TP=2
sudo bash scripts/report.sh --full > my-rig-tp2.txt

run TP=4
sudo bash scripts/report.sh --full > my-rig-tp4.txt

The reports in our native format is something that would really help. Whenever you get a moment, no rush.

[shaxs]

Thank you! @shaxs could you please submit rig reports for both TP=2 and TP=4?

run TP=2
sudo bash scripts/report.sh --full > my-rig-tp2.txt

run TP=4
sudo bash scripts/report.sh --full > my-rig-tp4.txt

The reports in our native format is something that would really help. Whenever you get a moment, no rush.

Yes I will do so. My results above were in reference to using https://github.com/DominikBucko/qwen38-flash-next-2x3090

I'm currently trying to run the 3090 q3 and q4 dual GPU and quad GPU tests on my rig but I'm running into thermal throttle issues on a few cards for some reason. Trying to troubleshoot.

[Whamp]

I made a custom mtp implementation with INT4 experts to save KV Cache space.

Pretty good results:

C1 89.67 tok/s +32.8%
C2 146.85 tok/s +24.6%
C4 243.11 tok/s +17.5%

https://huggingface.co/hampsonw/Qwen3.8-Flash-Next-W4A16-AutoRound-MTP-INT4-Experts#results-on-my-hardware

317,179 KV cache tokens with MTP down from ~425,000 without.

Working on providing a recipe from my vLLM fork

[noonghunna]

great stuff @Whamp

[AntonProkopyev]

Native vLLM rig report: two RTX 3090 GPUs, EPYC 7K62, Derisked AWQ

Ran bash scripts/report.sh --full on September 6 against the native vLLM recipe linked in this thread.
The overall result is ADVISORY (exit 2). No correctness check failed, but the stress test flagged a thin VRAM margin.

Full native report and deployment notes: public gist.

Hardware / configuration	Value
CPU	AMD EPYC 7K62, 48 cores / 96 threads, one socket
RAM	373.3 GiB visible to Linux. EDAC reports 384 GiB across six populated channels
GPUs	2× RTX 3090 24 GB, PCIe Gen4 x16, no NVLink, 250 W limits
Target	davetha/Qwen3.8-Flash-Next-DERISKED-W4A16-AWQ
PLE	Original BF16 table in host RAM, approximately 95.37 GiB
Runtime	vLLM 0.1.dev20073+g8e685d198, Humming 0.1.12, TP2 + EP
Draft	Local INT4 group-32 MTP, depth 3
Expert cache	Dynamic LRU, 84 slots per layer per GPU, UVA CPU offload
KV / context	BF16 KV, 4,429,185,024 bytes per GPU, 262,144-token configured limit
Vision	Enabled, up to two images per request
Collectives	Custom all-reduce enabled, expandable_segments:False

The recipe uses the runtime overlays from DominikBucko/qwen38-flash-next-2x3090, with the newer launcher options.
The attached notes record the image digest and source revisions.

Measurements

Measurement	Mean	CV
Narrative decode	74.60 tokens/s	2.7%
Code decode	80.08 tokens/s	2.1%
Narrative / code TTFT	1.766 / 1.754 s
Prefill at 8,432 to 8,707 input tokens	1,350.63 tokens/s	11.2%
Prefill at 84,923 input tokens	1,679.49 tokens/s	0.1%

Decode used three warmups and five measured runs per prompt, with the standard 1,000-token narrative and 800-token code caps.
The sampler was temperature 0.6, top_p 0.95, top_k 20, and min_p 0.0, with thinking disabled.
Prefill used one calibration warmup and three measured runs per depth, with a fresh prefix for each request.
These are client-observed input-token/TTFT rates. The table uses actual token counts because calibration undershot the 10K/90K labels.
The report contacted vLLM directly, without LiteLLM in the measurement path.

Operational checks and limits

• verify-full.sh: PASS. The unused Genesis check was skipped.
• verify-stress.sh: all boundary and control-string checks succeeded, up to 209,683 actual input tokens. The 262K limit was not fully exercised.
• Stress advisory: 753 MB free VRAM, below the script's 1,024 MB agent-safety threshold.
• Continuous soak: PASS, five sessions with five turns each, zero request errors, zero empty replies, and 0 MiB VRAM growth from the warm baseline.
• Agentic test: PASS, twelve turns, reaching 36,346 input tokens. Warm-baseline TTFT was 2.365 s, and the final turn took 7.731 s.
• One agentic turn had an unmeasurable decode interval. Its TPS remains n/a in the report.

The uniform-haystack checks establish addressability, not retrieval quality on natural documents.
The separate quality packs and a matched engine/MTP/P2P A/B were not run.
The Derisked weights and Qwentium template also differ from the original checkpoint, so these numbers do not isolate an engine speedup.

Two reporting limitations are recorded in the notes: sudo authentication prevented DIMM-speed and privileged ACS collection, and automatic capture did not recognize this overlay's CPU-offload configuration.
EDAC supplied the channel topology separately. The launch flags and worker logs show that expert offload and MTP were active.