⚡ Qwen3.8-27B part 2 — the fast tier lands at 105 tok/s, plus NVFP4

[noonghunna]

Follow-up to #993, where we shipped five
Qwen3.8-27B slugs and deliberately withheld every number because the bench had run on a silently
degraded config.

Here are five more slugs — and this time, numbers.

vllm/qwen38-27b-dual-fast decodes at 105.01 tok/s on 2× RTX 3090. That is a 27B dense model,
262K context, on two consumer cards from 2020.

⚠️ Straight away, the honest qualifier: that is the code-shaped prompt. Narrative prose runs
72.4 tok/s, and its run-to-run spread is 13.9%, so treat the prose figure as indicative. Both
numbers come from the same canonical bench on the config as shipped.

---

The new slugs

Slug	Cards	Weights	Activations	Ctx	Port
vllm/qwen38-27b-dual-fast	2	Frozenlock AutoRound INT4	int8 (W4A8)	262144	8095
vllm/qwen38-27b-multi4-fast	4	″	int8	262144	8096
vllm/qwen38-27b-multi8-fast	8	″	int8	262144	8097
vllm/qwen38-27b-single-nvfp4	1	RadixArk NVFP4	16-bit	65536	8098
vllm/qwen38-27b-dual-nvfp4	2	″	16-bit	262144	8100

All 🧪 Experimental — --force required.

Only dual-fast and dual-nvfp4 have booted here. The 4- and 8-card slugs are
community-validated by design: this rig has exactly two 3090s, permanently, so the first person
to boot one is the validation. single-nvfp4 needs a 32 GB card and has never run anywhere.

---

The numbers — vllm/qwen38-27b-dual-fast

2× RTX 3090, TP=2, W4A8 + MTP n=4, fp8 KV @262144, stock vllm/vllm-openai:v0.27.1,
3 warm + 5 measured.

metric	value	CV
decode — code	105.01 tok/s	2.7%
decode — narrative	72.4 tok/s	13.9% ⚠️
prefill @10k	1780.5 tok/s	0.1%
prefill @90k	1308.1 tok/s	1.4%
TTFT (short prompt)	149 ms	0.7%
KV pool	499,666 tok (1.91×)	—
VRAM	20,616 MiB/card	—
verify-full	9/9	—
MTP acceptance	3.15	—

⛔ What we have NOT measured on this slug: 8-pack quality, NIAH fill depth, soak. No quality
claim is being made. Throughput without a quality number is half a picture and we would rather say
so than imply the other half.

---

Where the speed comes from

Two independent findings, each measured single-variable on this checkpoint.

1. W4A8 — int8 activations

Same INT4 weights, but the matmul runs on INT8 tensor cores instead of dequantising to FP16:

	W4A16	W4A8	delta
prefill @10k	1219.5	1728.9	+41.8%
TTFT @10k	8299 ms	5925 ms	−28.6%
decode — narrative	65.9	60.5	−8.1%
decode — code	61.1	56.3	−7.8%

⚠️ Decode gets slower. On the qwen3.6 tier W4A8 was ~free; here it costs ~8%. Treat "W4A8 decode
is free" as a 3.6 finding that did not transfer. It's a trade: buy prefill, pay decode.

It works on this checkpoint only because it is auto-round-packed [k/8, n], which is what our
vendored negative-scale fold assumes — a compressed-tensors sibling hard-fails the same patch.

W4A8=0 reverts to 16-bit activations.

2. MTP depth — the knee is n=4

Full sweep, one boot per rung, W4A8, 262K:

SPEC_N	narrative	code	accept len
0	62.59	57.45	—
1	52.66 (−15.9%)	64.15	1.94
2	66.63	80.63	2.71
3	66.67	84.53	3.35
4 ⭐	76.40 (+22.1%)	105.60 (+83.8%)	3.90
5	73.70	106.14	4.31
6	71.11	107.54	4.70

⚠️⚠️ Acceptance length never turns. It climbs to 4.70 at n=6 while throughput flattens — so
picking a draft depth from acceptance length selects n=6, which is wrong. Use throughput.

⚠️ n=1 is worse than no drafter at all on prose (−15.9%). Shallow speculation is not a safe
fallback on this model.

---

Things worth knowing before you launch

⚡ This slug draws more power than anything else in the catalog. W4A8 moves the bottleneck from
memory-bandwidth to compute, MTP n=4 raises the duty cycle, and TP=2 synchronises both cards so
their current spikes coincide. One user hit hard power-offs
(#1019) on a 1500 W supply — a multi-rail
one, where per-rail OCP trips long before the PSU's total rating. Combining to a single 12 V rail
fixed it. If you're on a split-rail supply or a marginal one, either combine the rails or run
W4A8=0. Note that a GPU power cap does not protect you here: caps average over roughly a
millisecond and the transients are shorter.

⚠️ MTP is exposed to vllm#50021 — an unbounded
index in the GDN spec-decode path that can kill the engine under sustained multi-turn traffic. It is
open upstream and still present in v0.27.1 (we bumped and re-checked). Mitigate with SPEC_N=0.
Our own reports: #758 /
#838.

⚠️ Concurrency is capped at 1 by default. W4A8 + MTP n=4 at two concurrent 16K prompts pushed
peak VRAM 1.75 GB over the budget and requests began failing. For concurrent serving set
SPEC_N=0 and raise MAX_NUM_SEQS — measured ceiling is ~49 tok/s aggregate at 10K context.

NVFP4 on Ampere is a compatibility path, not a fast one. dual-nvfp4 boots and passes
verify-full 9/9, but sm_86 has no FP4 tensor cores, so vLLM serves it through the Marlin W4A16
fallback — the boot log says so twice. We publish no TPS for it, deliberately: any number would
be a Marlin figure wearing an NVFP4 label. Its real case is Blackwell, and the first sm_120 boot is
the validation we actually want.

Thinking mode defaults to medium effort, not the template's xhigh. medium injects no
reasoning instruction at all — it is the un-nudged baseline. xhigh appends "think carefully,
validate key assumptions…", which is what makes it slow. ⚠️ high is not valid and raises at
template render; only xhigh / medium / low exist.

#993's llama.cpp numbers are still withheld. That re-bench on -ts 0.55,0.45 is still owed —
nothing here changes it.

---

👁️ Bonus: vision and video input work

Not planned, discovered while validating: the qwen3.8 slugs serve images and video with no extra
flags. Verified with colour and layout discrimination at temperature 0, and a 3.25 s clip whose
frame order came back correct — including the blended transition frame.

Video cost is linear and it's what bounds usable length (default fps=2):

resolution	tok/s of video	fills the 262K window in
128×128	23	190 min
256×256	71	~61 min
512×512	263	~17 min
768×768	583	~7.5 min

⭐ Hour-scale video fits the native 262,144 window at 256×256 — no rope extension needed.

Input only: it consumes media and emits text. Smoke test only — no vision quality gate.

---

Getting it

WEIGHT_KEY=qwen3.8-27b:autoround-int4  bash scripts/setup.sh qwen3.8-27b   # fast tier, 18.2 GiB
WEIGHT_KEY=qwen3.8-27b:nvfp4           bash scripts/setup.sh qwen3.8-27b   # NVFP4,     20.4 GiB

Run it

bash scripts/switch.sh --force vllm/qwen38-27b-dual-fast     # 105 tok/s code · 262K · :8095
bash scripts/switch.sh --force vllm/qwen38-27b-dual-nvfp4    # NVFP4, Ampere = fallback · :8100

Useful env: W4A8=0 (16-bit activations) · SPEC_N=0 (no drafter) · ENABLE_THINKING=true
· MAX_NUM_SEQS=N.

Run the evals yourself

Two non-overlapping passes: behavioural quality (the 8-pack) and operational health. Everything
below is vLLM — all five slugs in this post are vLLM, so unlike #993
there is no engine branching.

One-time setup — three of the eight packs run in Docker sandboxes that build once:

pip install git+https://github.com/noonghunna/benchlocal-cli.git
git clone https://github.com/noonghunna/benchlocal-cli
bash benchlocal-cli/tools/build-sandboxes.sh     # ~30 GB free; prune if tight

Without the images the 5 deterministic packs still run; the 3 sandboxed ones skip with a warning.

⚠️ If you have an older version of benchlocal-cli, reinstall and rebuild the sandboxes anyway. The current latest version - v0.9.9 - build date 2026-08-18 contains fixes for the verifier of some test packs, notably dataextract-15 and hermesagent-20. You may get incorrect or lower scores if using older versions.

Quality — both reasoning legs

Instruct is the shipped default, so leg A is the bare launch — no env vars.

⚠️ Each leg needs its OWN boot, in both directions. The container persists between runs, so
going back to instruct after a thinking run needs a reboot too. Skip it and the "instruct" leg is
silently a second thinking leg: both arms score alike and the A/B reads as a clean null. benchlocal
catches this — per-pack thinking_validity in the saved JSON.

# leg A — instruct (default; no env)
bash scripts/switch.sh --force vllm/qwen38-27b-dual-fast
bash scripts/quality-test.sh --full --no-thinking --sampling-from-server \
  --max-tokens 4096 --thinking-max-tokens 16384 --timeout-per-case 600

# leg B — thinking  ⚠️ all FOUR vars, or you run reasoning with INSTRUCT sampling
ENABLE_THINKING=true TEMP=1.0 TOP_P=0.95 PRESENCE_PENALTY=0.0 \
  bash scripts/switch.sh --force vllm/qwen38-27b-dual-fast
REASONING_EFFORT=low BENCHLOCAL_MODEL_TURN_TIMEOUT=900 \
bash scripts/quality-test.sh --full --enable-thinking --sampling-from-server \
  --max-tokens 4096 --thinking-max-tokens 16384 --timeout-per-case 600

⚠️ ENABLE_THINKING flips the template only — the sampler does not follow it. Miss the three
sampler vars and you run reasoning at presence_penalty 1.5, which the model card warns causes
language mixing. Tracked in #1014; when that
lands this collapses to one variable.

Any slug substitutes: vllm/qwen38-27b-{dual,multi4,multi8}-fast · vllm/qwen38-27b-{single,dual}-nvfp4.

Why each flag

These aren't decoration — each exists because its absence produced a wrong number for us:

Flag	Why
--sampling-from-server	Uses the compose's card-correct sampler instead of the pack's temperature=0. Without it you measure greedy decoding, not the shipped config.
--max-tokens 4096	The ~1024 default silently truncates long answers into token_limit failures that look like wrong answers.
--thinking-max-tokens 16384	The thinking arm needs the headroom.
--timeout-per-case 600	16,384 tokens takes real wall-clock. Too tight a cap produces timeout failures that read as content misses.
REASONING_EFFORT=	Env only — forwarded as the per-request OpenAI reasoning_effort, so it is the caller's choice. Since #1029 these composes also default to low server-side, but pin it anyway so the run records what it measured. ⚠️ Effort is not comparable across runs. ⚠️ high raises on vLLM; only xhigh/medium/low exist.
--repeat 3	Add it for anything you'll quote — with --sampling-from-server both legs are sampled, so single draws aren't quotable.

⚠️ Read the failure modes before reading the score. timeout and token_limit are harness
artifacts; only verifier_fail / wrong_answer are the model. A score that looks catastrophic is
often a budget that was too small. Note also that benchlocal retries model verdicts 3× by
default but does not retry timeouts — so a timeout row is a single sample against a clock.

⚠️ cli-40 has a second, independent timeout — set BENCHLOCAL_MODEL_TURN_TIMEOUT=900 on the
thinking leg. The two govern different layers:

knob	governs	default	scales with TPS?
--timeout-per-case	per-scenario HTTP call, harness → server	pack metadata	✅ startup decode-TPS probe
BENCHLOCAL_MODEL_TURN_TIMEOUT	one runner-owned sandbox model call (cli-40, hermesagent-20, bugfind-15)	300 s flat	❌ no

Because the sandbox cap is flat, whether you hit it is arithmetic: one 16,384-token thinking answer
takes 16384 / decode_TPS seconds, so anything slower than ~55 tok/s busts it. At this slug's
105 tok/s you have room; on a slower rig you do not.

Operational health

bash scripts/report.sh --full     # verify + stress + soak + bench + agentic, redacted, ~43 min

Don't pair that with rebench-full.sh — they re-run the same gates.

What'd help

• A 4- or 8-card boot. multi4-fast / multi8-fast have never run. First boot is the validation.
• A Blackwell NVFP4 boot. Everything we've measured on nvfp4 went through the Ampere fallback.
• A 32 GB single card for single-nvfp4 — never booted, anywhere.
• 8-pack quality on any of these. We have throughput and no quality numbers.

Report via numbers-from-your-rig, or bash scripts/report.sh --full for a redacted paste-ready
bundle.

Credits

• @Qwen — the model.
• Frozenlock — the AutoRound INT4 quant the
  entire fast tier runs on.
• RadixArk — the NVFP4 export.
• @Zippo2000 (#1019) — for the multi-rail
  PSU finding, which is in the launch notes above because of that report.

[chriskerley78910]

Am legit wanting to try and run the 8-pack soon. Have not yet at because in the past had issues getting it to run.

Was using an INT8 quant of Qwen3.8 just before your PR. Just did a quick bench comparing the two. It performs just like Qwen3.6 did on my 2x3090s (which is too be expected). Its nice to have that big prefill boost back again :)

qwen3.8-27b — int4 (w4a8) vs int8 (w8a16)

Role	Model	Link
Ampere Accurate · Slow	Qwen3.8-27B-INT8-W8A16-MTP	model link
Ampere Fast	Qwen3.8-27B-INT4-W4A8-MTP	club3090 fast teir model
Ampere Accurate · Fast (proposed)	Qwen3.8-27B-INT8-W8A8-MTP	? — would combine int8's accuracy with int4's w8a8 prefill speed

Each table compares one factor between the two runs.

Headline: int4 wins on every throughput axis.
Decode is ~12–27% faster; prefill is ~48–72% faster (int4 processes the same
prompt in roughly half the time).

⚠️ Caveat: the int8 run logged a swap check FAIL (531 MB of the serving process
paged out), so its numbers are suspect and int4 likely looks relatively better than a
clean int8 run would.

Decode speed

Mean decode tokens/s (measured runs). int4 leads by a wide margin in both workloads.

Factor	int4 (w4a8)	int8 (w8a16)	Δ int4 vs int8
Decode · Narrative (1000 max tok)	98.36 tok/s	84.56 tok/s	+16.4%
Decode · Code (800 max tok)	137.99 tok/s	109.11 tok/s	+26.5%

Prefill speed

Mean prefill tokens/s (client-side, cache-busted). The gap is far larger than for decode.

Factor	int4 (w4a8)	int8 (w8a16)	Δ int4 vs int8
Prefill · 10K prompt tok	3259.66 tok/s	1891.57 tok/s	+72.3%
Prefill · 90K prompt tok	2115.79 tok/s	1428.49 tok/s	+48.1%

Notes

• Decode/prefill are the client-side summary means (5 measured runs for decode, 3 for prefill).
• Prefill 10K gap ≈ 1.73× (3259.66 vs 1891.57); prefihttps://docs.github.com/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntaxll 90K gap ≈ 1.48× (2115.79 vs 1428.49).
• Decode narrative gap ≈ 1.16×; code gap ≈ 1.27×.
• int8 swap check: FAIL · int4 swap check: PASS.

[sunhaozhe-glitch]

能支持一下这个作者的项目么，他量化的版本真的很好：DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-MTP-GGUF

[noonghunna]

yes of course. We already have one of his old deckard quant in our catalog. I will include his newer work in due course.

[sunhaozhe-glitch]

非常感谢您所做的出色工作！作为您的一名中国粉丝，我十分认可您的诸多宝贵贡献，期待这款新模型能够被收录。

[trial123Zel]

A 32 GB single card for single-nvfp4 — never booted, anywhere.

#1047

[sunhaozhe-glitch]

老哥来活了，Ornith-1.5-35B-A3B，这个模型有点厉害哦，有ornith-ai/Ornith-1.5-35B-A3B-MLX-4bit，您有空给支持一下vllm呗。

[opteemister]

Ran the super/ultra tiers for three days on 2× RTX 3090 @250w (PCIe, no NVLink) under real agent traffic — bare venv v0.27.1 + the vendored backport applied manually, Frozenlock target, drafter n=7, thinking-low default, temp 1.0.

The tier table above holds at shallow context but inverts with depth. Same server, same day, decode tok/s by KV depth:

Depth	ultrafast (bf16 KV / FA2)	superfast (fp8 KV / FlashInfer)
~5K	105–220 (peak 220, AL 6.3)	80–120
~40K	52–92	80–120
~165K	12–18	50–80

At 165K the drafter is still healthy (AL 3.0–4.3) — the forward itself slows to ~4 steps/s (~230ms/step). Suspect FA2 × this model's head_dim 256 on sm_86 — which would also explain why the backend list doesn't even offer FLASH_ATTN when fp8 KV is set. If right, the super/ultra crossover sits around 20–40K, so ultra is a fresh-short-chat tier; agent workloads live past the crossover. Might deserve a header note.

Also validated on superfast: max_num_seqs=2 with the boot-log kv-cache-memory suggestion (pool 460K) serves 200K+50K concurrent decode at 107 tok/s aggregate; prefix caching in mamba align mode is exact and turns turn-2 of a 150K chat into seconds (the "misses" I chased were a client mutating the prompt head). One HOL note: at max_num_batched_tokens=8192 a cold 165K prefill starves the co-running decode to ~0.2–1 tok/s (one token per ~5s chunk); 2048 + long_prefill_token_threshold=1024 keeps it visibly alive.

Venv gotchas for anyone else off-docker: patch_topk_4arg_compat.py hardcodes the container path (edit T), and don't launch from inside site-packages/vllm — vllm/tokenizers shadows the tokenizers lib via cwd and the DFlash2DraftModel registry inspection dies on a confusing circular import.

Frozenlock: no acceptance collapse through ~50K+ cumulative gen tokens here.