Qwen3.8-27B on one RTX 3090

Serving setup for Qwen3.8-27B on a single 24 GB consumer GPU with vLLM — 150k token context and an OpenAI-compatible API with key auth, in two ready-made modes.

The image is prebuilt and pushed to ghcr.io on every commit — the build applies all patches/ and runs verify.sh as its gate, so latest is always the current stack. The first start pulls it (9.5 GB), downloads and requantizes the model (~20 GB, once, into ./models), and serves on port 18020. Pick a mode — one GPU serves one at a time:

git clone https://github.com/syv-ai/qwen38-27b-rtx3090 && cd qwen38-27b-rtx3090

docker compose --profile single up -d    # one or a few people chatting
docker compose --profile batch  up -d    # API backend, many concurrent requests

Both modes share one install — the mode is just which launch script you run. Speculation wins below ~8 concurrent users on short prompts, plain batching above; on long independent sessions the crossover is much earlier, because a speculating request reserves recurrent-state pages the pool has few of — the concurrency paragraph under "DFlash2 at 240k" has the measurement. Numbers are vllm bench serve on an RTX 3090 at a 250 W power limit. If the card is yours alone, the fastest configuration is three environment variables away: If you are the only user.

Prefill is a separate budget from either: ~1,810 tok/s at 1k inputs in batch mode (~1,210 single-user), ~1,000 tok/s at 100k, so a 100k prompt costs ~100 s of TTFT (full matrix). How each number was won: docs/optimizations.md.

The server listens on 0.0.0.0 and is unauthenticated unless you give it a key. For anything past your own machine, add one first — everything reads it from .env or api_key.txt, and nothing needs it otherwise:

echo "VLLM_API_KEY=$(openssl rand -hex 24)" > .env

No compose, no clone — plain Docker runs the same image with one command and prepares the model itself on the first boot (into a named volume, so it survives container replacement):

docker run -d --name qwen --gpus all --ipc=host -p 18020:18020 \
  -v qwen-models:/app/models -v qwen-cache:/cache \
  --restart unless-stopped ghcr.io/syv-ai/qwen38-27b-rtx3090:latest

batch after the image name is the other mode, and the knobs compose reads from .env become -e flags (-e VLLM_API_KEY=..., -e SPEC=dflash2, ...) — docs/docker.md has the mapping.

Or by hand in a venv (same steps: model download, requantization, vLLM patches, verify.sh) — see Setup.

The command above starts the conservative default — MTP speculation, 8 request slots, 64k context, 120 tok/s greedy at C1. Two settings are worth more than every other knob in this repo put together, and a third is worth a great deal on one particular workload:

printf 'SPEC=dflash2\nPREFIX_CACHE=1\n' >> .env
# add DFLASH_TOKENS=15 if your answers quote your prompts — see below
docker compose --profile single up -d

or, in the venv install:

venv/bin/python prepare/fetch_dflash2.py   # once, 1.2 GB (Docker's prepare step does it for you)
SPEC=dflash2 PREFIX_CACHE=1 bash single-user/start_qwen.sh

SPEC=dflash2 swaps Qwen's MTP head for the DFlash2 block drafter: 7 tokens proposed in one pass instead of 4 chained ones. DFLASH_TOKENS=15 then lets the target verify 16 tokens per step — the drafter still proposes the 7 it was trained for, and the remaining positions are filled from the request's own context, which costs nothing to draft and is exactly right whenever the answer quotes the prompt. PREFIX_CACHE=1 keeps the document you already sent, both its attention KV and its recurrent state. One request at a time, greedy, RTX 3090 at 250 W:

VLLM_DFLASH2_CHAIN=1 adds drafter-free n-gram chains on top (#38, ported from @Dmtrii-tesla's fork with permission): while a request keeps reproducing its context, whole verify blocks come from history alone and the drafter's forward and graph replay are skipped until the first rejected token — +7% on the copy cell here (256.9 → 276 tok/s at DFLASH_TOKENS=7), flat on prose, greedy requests only by default (patches/dflash2-ngram-chains.patch explains why sampling keeps the drafter). Off by default.

_{* drafting from the context only exists in SPEC=dflash2. The two right columns are one server session, where run-to-run greedy divergence is ±3-5%; reproduce them with venv/bin/python bench/labd_bench.py <tag> --ctx 20000.}

PREFIX_CACHE=1 is orthogonal to the other two and worth as much again in a chat client: a second turn against that same 25k-token document takes 0.56 s to first token instead of 22.4 s, with the answers unchanged token for token.

Read that table by column, not by its last cell. SPEC=dflash2 is the upgrade for everyone; DFLASH_TOKENS=15 is for one workload. On chat it is worth 1%, because the eight positions past the drafter's own block are filled from the prompt and a chat answer does not quote the prompt — measured over bench/prompts_real.jsonl, positions 7-14 take 72 of 11,069 accepted tokens (0.65%), and @changtimwu measured exactly zero for them on a TP=2 box in #22. What you pay for that 1% is half the request slots and 8k of context, because a 16-token verify block doubles the recurrent-state page every resident request holds (1.66 GiB against 0.88 by the gotcha-33 fit). So: set it if you are quoting documents or applying edits, where it is worth 47%, and leave it at the default 7 for a chat or agentic client. DRAFT_TOKENS/DFLASH_TOKENS is one variable you can flip per service.

All of it is lossless: speculative decoding samples the same distribution as no speculation at all, the prefix cache resumes recurrent state rather than approximating it, and GSM8K reads 96.0-96.5% across the three columns. SPEC=dflash2 is a one-user mode either way (see concurrency). Every other knob: single-user/.

bash kvarn/install.sh                # applies kvarn-v2-runner.patch as its second stage
SPEC=dflash2 CTX=huge PREFIX_CACHE=1 bash single-user/start_qwen.sh

Where CTX=long doubles the DFlash2 pool with int8 KV (138k), the KVarN cache takes the same idea further: 268k tokens of pool at 245760 max-model-len, on the same pinned budget. No kernel work — the KVarN Triton kernels run unmodified on the V2 runner; the seven fixes in kvarn/kvarn-v2-runner.patch are allocator and geometry logic (the patch header walks through them, including an upstream vLLM bug in the mamba align resume path, and a NaN path in the DFlash2 candidate selector that KVarN noise exposes on verbatim-reproduction content). Two machines, both RTX 3090 at 250 W, bench/labd_bench.py --ctx 20000 — the contributor's WSL2 box and this repo's bare-metal one, which do not agree on decode rate and do agree on everything else:

_{Context for the GSM8K column: every configuration this repo already ships reads 95.0-96.5% on the same 200-question harness (docs/quality.md), and 95.0% is the batch-mode default. 95.2% at n=600 (±0.9 points) therefore sits inside the band rather than below it — which is the useful comparison, since this mode inherits KVarN's lossy 4/2-bit cache and should be judged against the other lossy configurations rather than against bf16. Repeat runs of the reproduction check on bare metal are bit-identical (same step count, same 1,150 characters), which is the property that was missing before PIECEWISE — see below.}

One caveat to the "all of it is lossless" paragraph above: the speculation here is still exact, but this mode inherits KVarN's 4/2-bit KV cache, which is lossy — the same trade CTX=huge already makes (deep-needle retrieval passes at 200k).

On WSL2, every SPEC=dflash2 profile needs VLLM_WSL2_ENABLE_PIN_MEMORY=1 — not just CTX=huge. The drafter's architecture forces vLLM's V2 model runner (_is_dflash2_draft() in config/vllm.py), the V2 runner allocates UVA buffers before the weights load, and vLLM leaves pinned memory off by default under WSL2, so a clean venv aborts with RuntimeError: UVA is not available before it prints anything model-shaped. Those buffers work fine on the paravirt driver. Note the name: VLLM_WSL_PIN_MEMORY is not a vLLM variable and setting it does nothing — this README named it for 22 minutes on 2026-08-21 (589daae, fixed in 27f51fa), so a tree cloned in that window will have it.

One knob this mode used to set for you, and now sets only for MTP: cudagraph_mode=PIECEWISE. Prefix caching and a captured (FULL) verify step did not mix on this path. On WSL2 that showed up as acceptance collapsing to about one token per step; on bare metal it also corrupted the output — special-token ids leaking into the stream, 1 of 1,176 characters matching the source instead of all of them. It is the capture rather than the drafter: eager is clean, LOOKUP=0 is not, forcing a fixed verify-block length is not, and PIECEWISE — which keeps the compiled graphs and leaves only the multi-query verify uncaptured — restored both the speed and the correctness on both machines.

dflash2 has its FULL graphs back (a75ee4b fixed the residue, b356e31 then swept all 128 residues under FULL with 0 broken), so at HEAD SPEC=dflash2 CTX=huge PREFIX_CACHE=1 runs captured. SPEC=mtp CTX=huge still forces PIECEWISE, and that one is a correctness constraint rather than a preference: under FULL it breaks at one prompt length in 128 and nobody has fixed it. Either way prefix caching stays on, which is what the mode is for — turn 2 over a cached 100k document costs 4.7 s against 169 s cold.

CUDAGRAPH_MODE=FULL_AND_PIECEWISE overrides the MTP line for anyone hunting the root cause. Treat that as unsafe rather than merely slower.

What that trade costs, re-measured at HEAD. The numbers this README used to carry here had FULL_AND_PIECEWISE at 38 tok/s (1.97 per step) on the 25k copy task against PIECEWISE's 132, and called it 3.5x. That was not the capture mode — it was the residue bug, which a75ee4b fixed. With the same server and only the capture toggled (bench/labd_bench.py --ctx 20000, SPEC=dflash2 CTX=huge PREFIX_CACHE=1, decode tok/s):

They are the same. Five of the six are within 2%; quote differs by 13% in PIECEWISE's favour, which is greedy divergence on the task that diverges most, and it is what puts PIECEWISE 3% ahead overall. So at this context length the capture mode is not a performance decision at all, in either direction.

Short prompts are where a difference was measured, and that measurement is older: 78/125/202 tok/s captured against 74/102/176 piecewise on de/en/code, i.e. 13-18%. Treat that as an upper bound — @mjungnickel18 measures 0.2-2.3% for the same comparison when only runs with identical step counts are compared, and he is right that greedy runs which take a different number of steps are not comparable. Past 8k the two are within noise on bare metal (111.8 vs 109.3 tok/s at 8k, 78.2 vs 86.1 at 16k, 68.9 vs 73.3 at 32k, 58.4 vs 56.0 at 50k, unique prompts, one server per mode). Under GPU passthrough on a VM the same comparison costs 2-3x, reported in #13 and consistent with the uncaptured verify being launch-bound: launches that are nearly free here are not free there.

Two limits worth knowing before you point this at anything: what it does with more than one user, and what the long verify block costs.

It is a one-stream mode, and the limit is the pool rather than MAX_SEQS. An earlier version of this paragraph said the knob was the seat count and that MAX_SEQS=8 lifts it. It does not, and @mjungnickel18 was right to push back in #25. A resident request reserves 1+k = 8 recurrent-state slots — 15.8% of the 69,758-token CTX=fast pool, ~0.82 GiB of its pinned 5.20, which is the 0.88 GiB gotcha 33 fitted from the memory model — before it holds one token of context. Seven fit with 128-token prompts, five with 4k-token ones and two with 16k ones. MTP's k=4 costs 0.44 GiB, so eight fit, four of them at 16k. Past that the extras queue, and once the pool is full something has to be preempted and recomputed to make room — a ramp of tiny requests hits that at the seventh. MAX_SEQS decides how many requests are admitted, not how many can run. The pool itself is unchanged by the setting (268,169 tokens at CTX=huge either way, ~8 MiB total between 1 slot and 8 — gotcha 33), which is the part of the old paragraph that was right.

What concurrency actually costs, measured with bench/conc_ladder.py on distinct 4k-token prompts — each salted so nothing is served out of the prefix cache — 256-token answers, MAX_SEQS=8, CTX=fast, 250 W:

Two things to read off it. The verify step does batch — aggregate throughput keeps climbing for both speculators, and no preemption happens at any of these points — so "the block verify does not batch" is not what is going on. What it costs is latency: each additional resident request adds about 7 ms to every forward pass under DFlash2 (5 ms under MTP), so the second user roughly halves your tokens per second and the fourth roughly quarters them. That 7 ms is not attention over their context — with 128-token prompts the slope is the same (25.2 → 45.8 ms from one resident to four) — it is the per-request recurrent state, the same thing that limits residency. And MTP keeps scaling to 8 streams where DFlash2 runs out of pool at 5, which is the whole of its C8 advantage. Point one person at DFlash2; point a team at SPEC=mtp or batch mode.

On CTX=huge, raising MAX_SEQS is worse than not raising it. That profile defaults to 2 seats, and the reason is the same state page against a smaller pinned pool (4.90 GiB): five residents with a short prompt, four with a 16k one. Force it to 8 and feed it eight independent 16k-token streams and the scheduler starts evicting — 10 preemptions in one run, peak occupancy 99.4%, per-stream 3 / 7 / 72 tok/s, end-to-end aggregate down to 10.3 from 13.0 at a single stream. The same eight streams against the shipped 2 seats: 0 preemptions and 14.4 tok/s, i.e. 40% more work done by admitting fewer requests. An earlier version of this README recommended exactly that override. Leave the seats where they are.

Make the streams long and independent and it stops being about decode at all. Eight 16k-token prompts with nothing shared between them: end-to-end aggregate 15.8 tok/s against 131 for a single stream, mean TTFT 71.7 s, two requests resident — but the decode-only aggregate over the same run is 183 tok/s, tokens per step is unchanged at 3.71 and nothing is preempted. The run is 131k tokens of prompt at ~1,600 tok/s and 2,048 tokens of answer, so it is a prefill measurement wearing a decode measurement's units, and SPEC=mtp reads the same 15.0-15.9 there. If your clients each bring their own long document, that is the number you get, and no speculator changes it.

What does change it is sharing the document. The same eight 16k streams with one shared prefix and PREFIX_CACHE=1 (bench/conc_ladder.py --shared):

One stream pays the prefill; everyone after it hits the cache, and four concurrent readers of the same document get 148 tok/s end-to-end against 15.9 when the documents differ. That is the shape of workload this mode is for — a chat client or a coding front-end against one codebase — and it is nearly a 10x difference from the same server on the same prompt length.

DFLASH_TOKENS=15 doubles the state page to 1.66 GiB: three residents with an empty context, two with 4k-token prompts, against five. That mode is single-user in the literal sense, and its launcher default of MAX_SEQS=4 is already the tighter number — do not raise it. DFLASH_TOKENS=15 MAX_SEQS=8 used to boot, answer /health and then die on the first concurrent batch (torch.OutOfMemoryError in the engine, every request 500); the launcher now caps the captured-graph size so that configuration degrades to piecewise instead (gotcha 38). It does boot at 240k since 82bd62d, which caps max_model_len to 221,184 above 7 drafts rather than letting the server fail to come up.

The capture mode is fixed at boot, and for SPEC=mtp CTX=huge the trade is not optional. What FULL does there is corrupt one prompt length in every 128, and only for a request that hits the prefix cache. The broken residue is a function of the draft count (R = 117 + k, the same for both speculators), which is why scoping the workaround to dflash2 was wrong the first time — SPEC=mtp CTX=huge shipped with the same bug at residue 4. Piecewise costs MTP nothing measurable (87.8/86.1/70.4/63.5 tok/s captured against 93.5/83.8/70.3/59.6 piecewise over 8k-50k), so it keeps PIECEWISE until residue 4 comes back verbatim under a full sweep.

Do not test that residue by its symptom. The location is deterministic and the damage is not: the same mtp residue has returned an empty answer, a one-character answer, and 400 tokens of fluent Danish inventing a task the prompt never asked for (2 of 1,146 characters matching the document). A detector keyed on "it repeats" or "it came back empty" passes at least one of those. Gotcha 37 in docs/gotchas.md has the residue table; bench/residue_sweep.py sweeps all 128 residues and judges every answer on how much of the document came back, and bench/verbatim.py self-tests that rule against all three shapes.

bash single-user/alternative.sh      # TRITON_ATTN + --kv-cache-dtype int4_per_token_head

Where KVarN reaches 268k with its own kernels, vLLM's stock int4_per_token_head cache now combines with the DFlash2 drafter too: 314,915 tokens of pool at 256000 max-model-len (1.23× concurrency) on one 24 GB card, no kvarn/install.sh. Three boot blockers stood in the way — a padded-page view error under the hybrid block-promotion geometry, and a causal-only assert plus missing per-seq-causal plumbing in the int4 Triton kernel, which the drafter's 8-row draft block needs (patches/int4-kv-per-token-head.patch, contributed in #42 by @lachhabw).

The trade: the Triton attention backend plus the per-step int4 unpack cost about 20% of decode against the shipped config on short prompts (~86 vs ~104 tok/s e2e on the same probe), and — unlike KVarN, which has GSM8K and 100k-needle numbers above — int4-KV quality at depth is unmeasured here. Tool calling round-trips correctly and the lookup lane works; treat the rest as experimental until someone runs the quality harness on it, which is a contribution this README will gladly take.

Everything here is written for one 24 GB card; multi-GPU goes through untouched via EXTRA_ARGS:

SPEC=dflash2 PREFIX_CACHE=1 EXTRA_ARGS="--tensor-parallel-size 2" bash single-user/start_qwen.sh

What the second card is worth is now measured, not assumed — #40 ran a controlled 1-vs-2×3090 A/B on this harness (same box, same install, PCIe 4.0 x8, no NVLink, 275 W):

• +16–35% decode across C1–C8 (DFlash2 greedy: 127.4 → 161.6 at C1). Worth it at batch 1: decode is bandwidth-bound here, and TP=2 is a second memory system, not idle compute.
• The DFlash2 residency ceiling is a 24 GB property, not a drafter property. On one card DFlash2 collapses at C8 (3.9 s TTFT — the recurrent-state pool exhaustion documented above) and MTP wins; on two cards DFlash2 wins at every concurrency measured. On 2×24 GB, point everyone at DFlash2.
• The launcher no longer pins KV_MEM under TP>1 (their finding, this fix): the pin is a single-card constant applied per worker, and it stranded ~16 GiB across two cards — 137,210 tokens of pool where GPU_UTIL sizing gets 302,223, at no measured decode cost. Export KV_MEM to pin anyway.
• Keep DFLASH_TOKENS=7 at TP>1 for now: the one T=15 datapoint at TP=2 (same issue) lost 27% at C1 with the lookup-filled tail accepting nothing, which is under diagnosis. The launcher warns.
• NVLink appears to buy little: #7's NVLink box at 330 W and #40's PCIe-x8 box at 275 W land within a few percent of each other at C1.

Also reported working: 2× RTX 5060 Ti 16 GB (#22) — the "would not fit on one card" case. The graph budget and MAX_SEQS defaults are still single-card calibrations; more A/Bs like #40's are the most useful numbers you can send.

Full tables per mode in batch/README.md and single-user/README.md; quality in docs/quality.md. Reproduce any of it with bash bench/run_benchmarks.sh batch|single against your own server.

ninfer-3090 is a standalone C++/CUDA engine that publishes cohort benchmarks for this model on this card. Theirs are 1,024-token answers from 29-34-token prompts, greedy, MTP3, int8 KV, prefix reuse off, an 8,192-token context window, and thinking on at reasoning_effort=medium, so their 1,024 tokens include reasoning. Ours are 8 realistic chat prompts (English, Danish, code), 1,024-token answers, model-default sampling, thinking off:

† measured before PREFIX_CACHE=1 became the single-user default. With prefix caching on, cached prefixes (and, under --mamba-cache-mode align, their recurrent-state pages) stay resident through the cohort ladder, so the DFlash2 residency ceiling bites earlier and this cell reads ~324 tok/s with a 2-3 s TTFT on the current stack — independently measured at 321.8 in #40, which is what prompted the re-measurement. C1–C4 read the same or slightly better than the table. One card, many concurrent users: SPEC=mtp remains the right mode.

Decode rate, C × 1000 / mean TPOT. All four of our columns were re-measured together on the current stack with bench/run_benchmarks.sh, keeping the second run after each restart as the script advises; greedy instead of default sampling reads 131.2 / 214.6 / 285.7 / 405.5 for DFlash2. Run-to-run spread on the same server is 5-8%, so treat one-decimal differences between the three right-hand columns as noise — C1 and C8 are where the modes genuinely separate.

Theirs is the decode column of their table; their end-to-end column reads 70.19 / 89.43 / 97.89 / 161.28, and an earlier version of this table quoted those against our decode rate, which was not like-for-like. What still is not like-for-like, in their favour and ours: their C1 is a single prompt in a single run with no error bars, thinking is on for them and off for us, and they publish no power limit or driver version — ours is an RTX 3090 pinned at 250 W. Peak VRAM is comparable (23.0 vs 22.1 GiB at C8). The gap is mostly vLLM's continuous batching plus the memory this repo's requantization frees up.

The whole stack is quantized, so the honest question is what it costs. Short version: IFBench 78.3 prompt-level strict vs 79.5 for the unquantized model (one point), perplexity 8.09 on 33k held-out tokens, GSM8K 96.5% (200 questions, greedy). Speculative decoding — MTP, DFlash2 and the lookup drafter — is exact by construction and changes none of it; the int8-activation steps in batch mode are the only knobs that trade accuracy for speed, and they cost 0.9-3.7% perplexity depending on how far you push them. Per-configuration tables: docs/quality.md.

Community reproductions of the single-user headline number, harness runs first. bench/run_benchmarks.sh single, greedy, second run (the first reads low):

Measured with their own clients rather than the harness — comparable to each other only loosely, and not rows for the table above:

• CMP 170HX 40 GB (GA100, sm80): 133.7 tok/s median (3x900 tok, greedy) on the shipped fast target — the first sm80 datapoint, level with the 3090 — and 97.8 tok/s on their own w8a16 int8 target after the sm80 repack workaround in #27 (gotcha 41).
• RTX 5090 32 GB (sm120): ~410-449 tok/s on code and ~198 on prose at CTX=fast, 500 W cap, roughly flat out to CTX=huge at 240k — different prompts, output length and rate definition, so deliberately not in the table (their own insistence, and correct). Setup gotchas and the full ladder: #35.
• Dual-GPU reports: the controlled 1-vs-2×3090 A/B in #40 (+16–35%, 161.6 C1 greedy at 275 W, PCIe x8 without NVLink; independently reproduced in-thread at 153.6/250 W by a second dual-3090 box), the NVLink dual 3090 in #7, dual 5060 Ti in #22. See "More than one GPU" above for what transfers.

Nine things, from requantizing both embedding matrices to drafting straight out of the prompt — one line each, then the reasoning and measurements, in docs/optimizations.md.

Measured cumulatively on the 3090, 64 concurrent, 128 in / 512 out, vllm bench serve random dataset:

And single-stream on realistic prompts (single-user mode, T = model default / greedy):

(Steps 4-6 are the same 8-prompt protocol; greedy is deterministic for a given server and request order but differs between configs and even with prefix-cache hits, so single runs carry ±3-5% on tokens/step — bench/run_benchmarks.sh single reproduces 111.1 / 120.0 tok/s decode at C1, the best repeats read 119 / 124.) Going deeper (k=5) loses again: 106 / 105. k=4 is the knee, but on vLLM 0.27.1's FlashInfer backend (needed for fp8 KV, i.e. for 150k context) four drafts crash the engine with an illegal memory access as soon as one request finishes while another is mid-generation — club-3090 reports the same "n=4 eventually dies, n=3 stable" pattern — so CTX=long drafts 3 and gives up ~7%; CTX=fast (FlashAttention, bf16 KV, ~64k context, the default) keeps k=4 and is also the only backend the split-KV attention patch applies to.

Two things that did not help, measured rather than assumed: fine-tuning the MTP head on the model's own outputs (KL halves, greedy top-1 on response tokens unchanged; drafter/README.md), and retuning Marlin's tile configuration for M ≤ 16 on sm86 (3-7% per GEMM in isolation, nothing measurable end to end — the remaining gap to peak bandwidth is the memory system's ramp on 16-92 MB reads, not the kernel).

The default install is the container (Quick start — the prebuilt image already contains everything this section builds), so this manual venv path is for hacking on the stack, or running it bare-metal. You need: a 24 GB Ampere or newer NVIDIA card, a recent driver, Python 3.12, ~40 GB disk. Everything below is CPU-safe to run while the GPU does other things; the container details live in docs/docker.md.

git clone https://github.com/syv-ai/qwen38-27b-rtx3090 ~/qwen-serving
cd ~/qwen-serving

python3 -m venv venv
venv/bin/pip install vllm huggingface_hub hf_transfer ninja \
  flashinfer-python flashinfer-cubin==0.6.13
# flashinfer makes the DFlash2 selector ~2x faster than its torch.topk fallback,
# and vLLM only *uses* it if nvcc is on PATH or flashinfer-cubin is installed --
# a bare `pip install flashinfer-python` silently falls back with one INFO line
# (#35). cubin publishes up to 0.6.13, so the version pair needs
# FLASHINFER_DISABLE_VERSION_CHECK=1, which the launchers export. Do not fix the
# mismatch by downgrading flashinfer-python: that drags torch back and breaks
# vLLM's C extension.

# model, ~19.5 GB
HF_HUB_ENABLE_HF_TRANSFER=1 venv/bin/hf download \
  dbirks/Qwen3.8-27B-W4A16-AutoRound \
  --local-dir models/Qwen3.8-27B-W4A16-AutoRound

# requantize lm_head + embeddings + the MTP draft module (CPU only, a few minutes)
venv/bin/python prepare/quant_lm_head.py models/Qwen3.8-27B-W4A16-AutoRound
venv/bin/python prepare/quant_embed.py   models/Qwen3.8-27B-W4A16-AutoRound
venv/bin/python prepare/quant_mtp.py     models/Qwen3.8-27B-W4A16-AutoRound
# 40k-token draft head for single-user mode (uses the shipped id list)
venv/bin/python prepare/build_draft_vocab.py models/Qwen3.8-27B-W4A16-AutoRound \
  --ids prepare/draft_vocab_ids.json
# single-user "fast" variant (~1 GB from the Hub, hardlinks the rest): int4-GPTQ
# lm_head + drafter; single-user/start_qwen.sh picks it up automatically
venv/bin/python prepare/fetch_fast_variant.py
# optional: the W4A16 DFlash2 block drafter (1.2 GB) for SPEC=dflash2 single-user mode
venv/bin/python prepare/fetch_dflash2.py

# patch vllm (all written against 0.27.1; reapply after upgrades)
for p in patches/*.patch; do
  patch -p1 -d venv/lib/python3.12/site-packages/vllm < $p
done
# optional: the KVarN 4/2-bit KV cache for 262k context (docs/long-context.md)
bash kvarn/install.sh

# api key — optional, but the server binds 0.0.0.0 and is open without one
openssl rand -hex 24 > api_key.txt

Then bash verify.sh --no-server — it checks the venv and vLLM version, that every patch in patches/ is actually applied, and that the model has been requantized (lm_head, embeddings, MTP module, draft head). Then pick a mode and follow its README:

• batch/ — throughput. bash batch/start_qwen.sh
• single-user/ — latency. bash single-user/start_qwen.sh

First start takes a few minutes (torch.compile, CUDA graph capture, flashinfer JIT). Test it:

curl http://localhost:18020/v1/chat/completions \
  -H "Authorization: Bearer $(cat api_key.txt 2>/dev/null)" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.8-27b",
       "messages": [{"role": "user", "content": "hej"}],
       "chat_template_kwargs": {"enable_thinking": false}}'

Qwen recommends temperature 0.7 / top_p 0.8 for instruct mode, and 1.0 / 0.95 with thinking enabled (the default).

Tool calling works over the same endpoint — send tools with tool_choice: "auto" and the reply carries tool_calls. Both launchers set --enable-auto-tool-choice --tool-call-parser qwen3_coder; the parser has to read Qwen's XML call format, which is what this model's chat template emits — not the JSON that hermes reads. TOOLS=0 turns it off.

To check the numbers on your own card: bash verify.sh (also probes the live server and prints which attention backend and KV pool it came up with), then bash bench/run_benchmarks.sh batch or ... single reproduces the tables above against the running server (--prefill and --long add the prefill matrix and the long-context rows), bash bench/real_rep.sh <tag> 3 0 repeats the single-stream row, and python bench/quality_battery.py <tag> the perplexity / GSM8K rows. For the concurrency rows, python bench/conc_ladder.py --n 1,2,4,8 --ctx-tokens 4096; for the prompt-length bug, python bench/residue_sweep.py <tag> (all 128 residues) with python bench/verbatim.py as its offline self-test.

Apache-2.0, same as the model.