⚡ Qwen3.8-27B part 3 — the DFlash2 tier hierarchy: super, ultra, across speed and fidelity

[noonghunna]

Follow-up to #1024, where the fast tier benches ~100 tok/s on 2× 3090.

This part isn't one more slug — it's a naming and a shape. Every Qwen3.8-27B tier now comes in three depths of speculation, on top of two weight classes:

vllm/qwen38-27b-dual-ultrafast decodes code at 231 tok/s on 2× RTX 3090. Same two consumer cards, same 27B dense model. That is 2.3× the fast tier — from a drafter, not a bigger GPU.

⚠️ The honest qualifier, up front: that is the code-shaped prompt, and it costs context. Narrative prose runs 128 tok/s; and to hit that decode speed the dual card drops from 262K context to ~200K. Both numbers are the canonical bench on the config as shipped. No quality number is attached yet (see the bottom).

---

The shape

Two weight classes — the speed series (int4) and the fidelity series (fp8) — each now in three drafter depths:

	base (built-in MTP)	super (DFlash2 + fp8 KV)	ultra (DFlash2 + bf16 / FlashAttn)
speed (int4)	fast	superfast	ultrafast
fidelity (fp8)	max	supermax	ultramax

• super* swaps the built-in MTP head for an external DFlash2 block-drafter while keeping the base tier's fp8 KV. You keep the full context envelope; you gain code throughput; and because the drafter is external, it structurally sidesteps the built-in-MTP acceptance-collapse class entirely.
• ultra* runs that same drafter on bf16 KV + FlashAttention — the fastest decode path on Ampere — at the cost of context (bf16 KV is 2× fp8, so it doesn't fit 262K on two cards).

One drafter serves all six DFlash2 slugs. It's keyed to the base architecture, not the weight quant, so the same syvai/Qwen3.8-27B-DFlash2-W4A16 drives both the int4 and the fp8 targets.

---

The numbers — 2× RTX 3090, TP=2, verify-full 9/9 each

Canonical bench (3 warm + 5 measured), stock vllm/vllm-openai:v0.27.1 + the vendored DFlash2 backport. Decode tok/s, narrative / code:

Speed series (Frozenlock AutoRound INT4)

Slug	Drafter · KV / attn	Ctx	narr / code	vs fast
dual-fast	MTP n=4 · fp8	262K	73 / 100	—
dual-superfast	DFlash2 · fp8 / FlashInfer	262K	78 / 141	+41% code
dual-ultrafast	DFlash2 · bf16 / FlashAttn	~200K	128 / 231	+131% code

Fidelity series (official Qwen FP8)

Slug	Drafter · KV / attn	Ctx	narr / code	vs max
dual-max	MTP n=3 · fp8	262K	69 / 87	—
dual-supermax	DFlash2 · fp8 / FlashInfer	144K	68 / 130	+49% code
dual-ultramax	DFlash2 · bf16 / FlashAttn	64K	90 / 172	+98% code

All 🧪 Experimental — --force required. multi4 / multi8 variants of every tier ship too, community-validated by design (this rig has exactly two 3090s, so the first person to boot one is the validation).

---

Where the speed comes from — and why ultra trades context

One finding drives the whole hierarchy, and it's a hardware wall worth stating plainly.

On Ampere (sm_86), FlashAttention and fp8 KV cache are mutually exclusive — and it's a real kernel limit, not a config we forgot to flip. vLLM's FLASH_ATTN backend defers a quantized KV dtype to a capability check that requires FlashAttention 3 (Hopper) or 4 (Blackwell). Our 3090s run FA 2, which has no fp8-KV read path, so the moment you set fp8 KV, vLLM routes attention to FlashInfer instead.

That splits the DFlash2 drafter (whose block attention wants FlashAttention) two ways:

• super* keeps fp8 KV → runs on FlashInfer → keeps the full 262K envelope, at a decode cost (~40% vs the FlashAttn path).
• ultra* keeps FlashAttention → must use bf16 KV → fastest decode, but bf16 KV is 2× the size, so context drops.

So the hierarchy isn't arbitrary — it's the two sides of one Ampere constraint. super = "all the context, collapse-immune, faster than MTP." ultra = "maximum decode, shorter context."

Why the fidelity duals cap lower

supermax (144K) and ultramax (64K) cap well below their int4 cousins. The FP8 weights are ~29 GiB, and the external DFlash2 drafter costs ~2.6 GiB of KV pool that the built-in MTP head doesn't. On two 24 GB cards that leaves little room — bf16 KV especially. The multi4 / multi8 variants recover the full 262K once the weights split across more cards.

---

For 4+ GPU rigs — where the fidelity tiers stop compromising

Every tier above also ships in 4-card (multi4, TP=4) and 8-card (multi8, TP=8) form — the same speed×fidelity × base/super/ultra matrix, twelve DFlash2 slugs in all. We can't bench them (the reference rig has exactly two 3090s), so they're community-validated by design — but if you have four or more cards, this is the topology the fidelity series was waiting for.

On two cards the FP8 tiers are VRAM-starved; more cards remove the compromise. The official FP8 weights (~29 GiB) leave so little room on 2× 3090 that supermax caps at 144K and ultramax at 64K — you trade context to keep FP8 precision. Split those weights across four or eight cards and the tradeoff disappears: FP8-precision weights and the full 262K context and the DFlash2 drafter, all at once. The dual fidelity tiers are the constrained version; multi4/multi8 is where full-quality FP8 finally gets the full window. (The int4 speed tiers already fit 262K on two cards — for them, more cards buy concurrency headroom, not context.)

The DFlash2 tiers available at 4-/8-card — all 🧪 (--force), no first-party numbers (the reference rig has two cards):

Slug	Series	Drafter · KV / attn	Context on 4+ cards
vllm/qwen38-27b-multi4-superfast	speed (int4)	DFlash2 · fp8 / FlashInfer	262K
vllm/qwen38-27b-multi8-superfast	speed (int4)	DFlash2 · fp8 / FlashInfer	262K
vllm/qwen38-27b-multi4-ultrafast	speed (int4)	DFlash2 · bf16 / FlashAttn	more headroom than dual's ~200K
vllm/qwen38-27b-multi8-ultrafast	speed (int4)	DFlash2 · bf16 / FlashAttn	more headroom than dual's ~200K
vllm/qwen38-27b-multi4-supermax	fidelity (fp8)	DFlash2 · fp8 / FlashInfer	→ full 262K (dual caps at 144K)
vllm/qwen38-27b-multi8-supermax	fidelity (fp8)	DFlash2 · fp8 / FlashInfer	→ full 262K (dual caps at 144K)
vllm/qwen38-27b-multi4-ultramax	fidelity (fp8)	DFlash2 · bf16 / FlashAttn	recovered (dual caps at 64K)
vllm/qwen38-27b-multi8-ultramax	fidelity (fp8)	DFlash2 · bf16 / FlashAttn	recovered (dual caps at 64K)

The base MTP tiers ship too: vllm/qwen38-27b-{multi4,multi8}-fast and -{multi4,multi8}-max. Context on 4+ cards is a VRAM-math projection except @A1RM4X's measured 240,635 (multi4 superfast/INT4 speed tier — see below).

There's already a strong 4-card datapoint. @A1RM4X ran the full gate on 4× RTX 3090 (#1064): verify-full 9/9, stress to 240,635 tok (91% of 256K) at 0 MiB growth, soak PASS, code decode 190 tok/s, both 8-pack arms (120/150 no-think · 128/150 think). ⚠️ That run was the INT4 speed tier on the incoai bf16 drafter (PR #1060), not the shipped syv-ai W4A16 — a strong cross-reference for the multi4-DFlash2 shape, not the exact shipped slug. The full-context-FP8 claim above follows from the VRAM math (weights split → room for 262K fp8 KV), not a first-party bench — a bench.sh on vllm/qwen38-27b-multi4-supermax from any 4-card rig is what turns it from projection into a shipped row. (MTP base tier: #1025 96/119, #1004 103/135 — same 240K ceiling.)

⚠️ Budget KV headroom for DFlash2. A1RM4X's other finding: the drafter's codebooks (~0.5 GiB/GPU) + 5-layer backbone cost VRAM the KV pool wants — pushing to 512K needed --gpu-memory-utilization 0.80 (0.90 OOM'd the first prefill). More cards give you the room; don't assume the MTP tier's gmu carries over.

Boot one, run report.sh --full, and file it via the numbers-from-your-rig template — that's what turns 🧪 into a shipped row.

---

Things worth knowing before you launch

⛔ No quality numbers yet. verify-full passes 9/9 on every dual slug, but the 8-pack, NIAH fill depth, and soak have not been run on any of these. Throughput without a quality number is half a picture — the c3 catalog leaves the 8pk column deliberately blank rather than imply the other half.

⚠️ Setup needs the drafter. The six DFlash2 slugs need the external drafter on disk. Fetch it with WITH_DFLASH_DRAFT=1 when you run setup.sh (it's ~1.2 GB, shared by every DFlash2 tier).

⚠️ Spec-decode is exposed to vllm#50021 — the same open GDN spec-decode index bug as the MTP tiers, still present in v0.27.1. Our collapse soak ran crash-free to 21k tokens, but treat that as "not observed," not "immune." Mitigate with SPEC_N=0.

⚠️ These decode numbers are a CUDA-13 ceiling, not a peak. Our image runs CUDA 13.0, where FlashInfer's radix top-k for the DFlash2 candidate selector can't JIT-compile and falls back to a slower sort — @oceanplexian found this in #1056. The selector's "sort-free" fast path also isn't in stock v0.27.1. So treat these figures as a floor for the config: a CUDA-12 build, or a vLLM carrying the sort-free selector path, should decode faster.

---

Launching these — the c3 cockpit

Every qwen3.8-27b slug — all four DFlash2 tiers × dual/multi4/multi8, plus the base MTP, single-card, and NVFP4 rows — is browsable and one-keystroke-launchable from c3, the terminal cockpit. Filter to 3.8, pick a row, press ⏎ to serve:

Install + launch (from the repo checkout):

uv pip install -e tools/serve-cockpit    # or: pip install -e tools/tui-core && pip install -e tools/serve-cockpit
c3                                        # launch  (also: python -m club3090_cockpit)

First run: press S → set your Model Dir + HuggingFace token → Ctrl+S, then r to browse the catalog. Filter to 3.8, select a slug, ⏎ serves it. c3 --lean hides the producer lane for a consumer-only view. c3 wraps the same launchers under the hood, so the DFlash2 tiers still need the external drafter on disk (WITH_DFLASH_DRAFT=1 — see "Things worth knowing" above).

Prefer the CLI? Every slug is also bash scripts/switch.sh --force <slug> — that's the form the eval commands below use.

Run the evals yourself

Same two-pass protocol as the fast-tier post (#1024) — behavioural quality (the 8-pack) + operational health. The flag-by-flag rationale (why --sampling-from-server, the two independent cli-40 timeouts, the failure-mode triage) is spelled out there and is identical here; below is the short form plus the one thing that's different: the DFlash2 slugs need the external drafter on disk — WITH_DFLASH_DRAFT=1 bash scripts/setup.sh (~1.2 GB, shared by all of them).

One-time setup — three of the eight packs build a Docker sandbox once:

pip install git+https://github.com/noonghunna/benchlocal-cli.git
git clone https://github.com/noonghunna/benchlocal-cli
bash benchlocal-cli/tools/build-sandboxes.sh     # ~30 GB free; prune if tight

⚠️ Reinstall + rebuild if you're on an older benchlocal-cli — v0.9.9 (2026-08-18) fixed the dataextract-15 / hermesagent-20 verifiers; older versions score wrong.

Quality — both reasoning legs, each its OWN boot

Instruct is the shipped default (leg A = bare launch). Going back to instruct after a thinking run needs a reboot too, or the "instruct" leg is silently a second thinking leg.

# leg A — instruct (default; no env)
bash scripts/switch.sh --force vllm/qwen38-27b-dual-ultrafast
bash scripts/quality-test.sh --full --no-thinking --sampling-from-server \
  --max-tokens 4096 --thinking-max-tokens 16384 --timeout-per-case 600

# leg B — thinking  ⚠️ all FOUR sampler vars, or you run reasoning with INSTRUCT sampling
ENABLE_THINKING=true TEMP=1.0 TOP_P=0.95 PRESENCE_PENALTY=0.0 \
  bash scripts/switch.sh --force vllm/qwen38-27b-dual-ultrafast
REASONING_EFFORT=low BENCHLOCAL_MODEL_TURN_TIMEOUT=900 \
bash scripts/quality-test.sh --full --enable-thinking --sampling-from-server \
  --max-tokens 4096 --thinking-max-tokens 16384 --timeout-per-case 600

Any DFlash2 slug substitutes: vllm/qwen38-27b-{dual,multi4,multi8}-{superfast,ultrafast,supermax,ultramax}.

Operational health

bash scripts/report.sh --full     # verify + stress + soak + bench + agentic, redacted, ~43 min

Don't pair it with rebench-full.sh — same gates.

What'd help

These slugs are all 🧪 — they have throughput and no quality numbers yet, so an 8-pack on any of them is what promotes it. Most wanted:

• A 4- or 8-card boot of the fidelity tiers (multi4-supermax / multi8-ultramax) — the full-context-FP8 claim above is VRAM math, not a bench.
• 8-pack quality on any DFlash2 slug, either reasoning leg.
• A CUDA-12 build — our CUDA-13 image caps the DFlash2 selector (see the note above); a CUDA-12 rig should decode faster.

Submitting is easy — and we want it even if the slugs DON'T boot on your rig:

bash scripts/report.sh --bench          # it works  -> the bench block (TPS + accept-len + peak VRAM)
bash scripts/report.sh > my-rig.md      # it doesn't -> a plain rig report (hardware + boot logs)

Drop either in a reply here or via the numbers-from-your-rig template. A rig report where a slug failed is as useful as one where it worked — a boot log + your GPU/driver/CUDA is exactly how we find the edges (CUDA-13 selector fallback, GPU-count, driver-branch mismatches).

Credits

This tier stands on community work — we used their submissions as the starting point, then shaped the slugs to our product hierarchy (a lean W4A16 drafter for more context; the speed×fidelity × base/super/ultra matrix):

• DFlash2 is upstream vllm-project/vllm#52816; the original block-drafter checkpoint is incoai/Qwen3.8-27B-DFlash2.
• @oceanplexian (#1056) shipped the first DFlash2 dual-3090 benchmark (120/218) and the vLLM boot fixes — and flagged the CUDA-13 selector fallback above.
• @A1RM4X (#1060) ported the 6-piece vllm#52816 overlay (the V2-speculator path that stops the checkpoint silently degrading to base DFlash) and ran the full validation gate — 8-pack, NIAH ceiling ladder to 240K, and soak — on the 4-card tier.
• @syv-ai backported the port to v0.27.1 and requantized the drafter to W4A16 (1.2 GiB) — the lean drafter these slugs ship.

Both community PRs used the incoai bf16 drafter (11.6 GiB, which caps context at 131K); we chose the W4A16 requant to keep more of the window. We vendor all of this — we don't own it.

---

Pick by shape: fast/max for the full 262K on a single stream; super* for +code throughput without giving up context; ultra* when you want the fastest decode and can live inside its context window. Speed series = int4 (smaller, faster); fidelity series = official FP8 (higher-precision weights).

[aklaiber82]

One configuration-safety note that may be worth making explicit in the tier docs:

For the current Qwen3.8 DFlash2 exports, SPEC_N is coupled to the checkpoint's dflash_config.block_size; it is not a free tuning knob. The DFlash2 implementation constructs the grouped-convolution block size as 1 + num_speculative_tokens, and _grouped_conv uses that value as its causal reset boundary. The published exports declare block_size = 8, so SPEC_N=7 is the matching depth.

A mismatch can load successfully because block_size does not shape any checkpoint parameter. It instead changes the causal boundary silently and may reduce draft acceptance rather than failing at boot. In particular, SPEC_N=15 should only be paired with an export that declares block_size=16.

Suggestion: treat SPEC_N as checkpoint-coupled metadata and validate 1 + SPEC_N == dflash_config.block_size at launch, preferably failing loud on mismatch.

[noonghunna]

thank you!

[paulp83]

#1085 [bench] 2x 5090 / qwen38-27b-dual-supermax
#1084 [bench] 2x 5090 / qwen38-27b-dual-ultrafast
#1079 [bench] 2x 5090 / vllm/qwen38-27b-dual-superfast
#1086 [bench] 2x 5090 / vllm-qwen38-27b-dual-ultramax