🐣 Incubating: Qwen3.8-27B — 5 slugs (1/2/4/8 cards, llama.cpp + vLLM), no numbers yet #993
Replies: 5 comments 2 replies
|
Oh yes!!! I am searching for a positive-symmetric INT4 quant to be able the INT8 activations flag. Missing the 50% boost in prefill speeds. |
|
focusing on stable serving than a faster one right now. Will do a slug on W4A16 once its available on https://huggingface.co/LeaderboardModel1 |
First boot numbers for
|
| GPUs | 2x RTX 3090 24 GB, driver 610.57.04, no NVLink |
| VBIOS | 94.02.26.48.5D and 94.02.42.00.F1 (two different AIB boards) |
| PCIe | GPU0 x2, GPU1 x16 |
| Power cap | 250 W both cards (defaults 350 and 370 W) |
| Board, CPU | B550 AORUS ELITE V2, Ryzen 5 5600X, 62 GB RAM |
| OS | Ubuntu 24.04.4, kernel 6.8.0-137, bare metal docker |
| Repo | e725b331 |
Only deviation from the compose: my GGUF lives outside the default MODEL_DIR, so I ran it with MODEL_DIR and GGUF_FILE overridden to point at it. Command line, image and everything else are as shipped.
Boot
It boots and serves. Two things worth reporting back:
VRAM lands exactly where the compose predicted: 22,504 and 22,574 MiB, against the 22504 / 22574 in the TOPOLOGY note. So the 0.55,0.45 split derived on your rig transfers to this one unchanged. n_slots = 1, n_ctx_slot = 262144, MTP draft context created, and retrying without pipeline parallelism did not appear.
The b10236 pin does load this GGUF. The compose lists engine compatibility as unproven ("that this specific GGUF loads on this specific build is an inference, not an observation").
It loads and serves, so that one can be closed.
scripts/bench.sh at 250 W
| wall_TPS | decode_TPS | TTFT | |
|---|---|---|---|
| Narrative (n=5) | 41.08 ±0.77 (CV 1.9%) | 42.49 ±0.80 | 791 ms |
| Code (n=5) | 48.55 ±1.32 (CV 2.7%) | 51.44 ±0.97 | 765 ms |
| Prefill | tok/s | TTFT |
|---|---|---|
| 10k (n=3) | 1166.15 ±6.95 (CV 0.6%) | 8.2 s |
| 90k (n=3) | 888.52 ±4.73 (CV 0.5%) | 102.6 s |
Draft acceptance over the measured window: mean 0.819, min 0.577, max 1.000, 7785 accepted of 11176 drafted, fired on 16 of 16 requests. That is at the compose's --spec-draft-n-max 2; I did not sweep it. The engine-side cross-check came out at 0.3% divergence from the client-observed rate.
scripts/soak-test.sh --continuous at 250 W
| verdict | PASS |
| boot / max VRAM | 45,140 MiB, growth 0 MiB of 200 |
| silent_empty | 0 of 25 |
| errors | 0 |
| p50 decode_tps | 63.22 |
| p95 TTFT | 8207 ms |
| tps_retention | 100.0% |
One pattern to note: median turn was 7963 ms, but 4 turns ran long, up to 24.6 s, and all 4 were turn 5, the last turn of a session. On those turns TTFT stayed normal while decode fell to roughly 40 tps against the 63.22 p50, so the extra wall time is going into decode rather than prefill, at the deepest accumulated context in each session. I saw the same thing on an earlier bare metal run of the same model (5 long turns, 4 of them turn 5), so it reproduced across two independent setups. Not thermal as far as I can tell: the first affected turn is session 1 turn 5, per-session medians show no upward trend across the run, and temperatures peaked at 65 to 67 C with HW Thermal Slowdown never Active.
Power cap, 200 W vs 250 W
Same container, same scripts, only the cap changed.
| Metric | 200 W | 250 W | change |
|---|---|---|---|
| Narrative wall_TPS | 36.98 ±0.56 | 41.08 ±0.77 | +11.1% |
| Code wall_TPS | 42.98 ±0.80 | 48.55 ±1.32 | +13.0% |
| PP tok/s | 938.6 | 1259.0 | +34.1% |
| Prefill 10k tok/s | 904.9 ±2.5 | 1166.2 ±7.0 | +28.9% |
| Prefill 90k tok/s | 665.7 ±4.6 | 888.5 ±4.7 | +33.5% |
| TTFT at 90k | 136.9 s | 102.6 s | -25.1% |
Prefill gains roughly three times what decode does from the same extra 25% of power. Memory clock sat at 9501 MHz in both runs and SW Power Cap was reported Active on both cards even at 250 W, so I assume the headroom is going into SM clock bound work. I did not go above 250 W because the repo's power cap unit documents that as this rig class's envelope.
On the BENCHMARKS efficiency measure (decode TPS divided by per card cap), 200 W gives 0.185 and 250 W gives 0.164, so 200 W remains the better efficiency point and is consistent with the existing 3090 rows. I only mention the prefill side because the 34 s of extra TTFT at 90k mattered more to me in practice than the decode difference, which is a statement about my workload rather than a criticism of the metric.
A note on the pin, since it may save someone a rebuild
Before using the container I had run the same config on bare metal llama-server built from source at 89e0aa6, which is 150 commits after the b10236 pin, with the compose flags transcribed by hand. The two agree closely:
| 250 W | bare metal 89e0aa6 |
pinned b10236 |
|---|---|---|
| Narrative wall_TPS | 41.05 | 41.08 |
| Code wall_TPS | 48.56 | 48.55 |
| Prefill 90k tok/s | 885.0 | 888.5 |
| Draft acceptance | 0.809 | 0.819 |
So for this model at least, the 150 commit gap does not appear to move the numbers, and the pin is not leaving anything obvious on the table.
Caveats
- The 250 W cap is user set rather than stock. Both bench and soak ran at it, and the 200 W arm is above for comparison.
- I did not run a NIAH ladder, so I cannot say how much of the allocated 262144 actually fills. The deepest thing exercised here is the 90k prefill probe. The compose already flags
this, and the Tess-4-27B sibling it points at fills to roughly 91%. - GGUF path was overridden via
MODEL_DIRandGGUF_FILEas noted above.
|
thanks for considering the upgrade to 27.1 as I have in my pr :-) Any reason not to keep vllm in sync ? |
➡️ Part 2 is up — and half the withheld numbers are now published#1024 — Qwen3.8-27B part 2: the fast tier lands at 105 tok/s, plus NVFP4 Five more slugs (3× AutoRound-INT4 "fast", 2× NVFP4), and this time with measured throughput:
Two things in this post are now out of date and corrected over there:
Also worth carrying across: these fast slugs draw noticeably more power than anything else in the |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
We've started incubating Qwen3.8-27B — five slugs across llama.cpp and vLLM, from a single card to eight. It's Qwen3.8-27B by @Qwen, served from the official FP8 checkpoint and from @unsloth's dynamic GGUFs. All credit for the model goes to the Qwen team; the GGUF quants are unsloth's — see Credits.
The headline: all five ship 🐣 Incubating with no performance or quality numbers attached — deliberately. We benched, then found the bench had run on a silently-degraded configuration, so we pulled the numbers rather than publish a floor of unknown depth. What we are shipping is a set of configs that boot cleanly, plus the traps we hit getting there.
The slugs
llamacpp/qwen38-27b-single-iq4xsllamacpp/qwen38-27b-dual-q8kxlvllm/qwen38-27b-dual-maxvllm/qwen38-27b-multi4-maxvllm/qwen38-27b-multi8-max(The single-card slug was
iq4nluntil 2026-08-19, when unsloth removed the IQ4_NL GGUF from the repo — it's nowiq4xs, and its default flipped toq4_0KV at the full 262K. Old launch names are dead.)The 4- and 8-card slugs have never run anywhere: our reference rig has exactly two 3090s, so they are community-validated by design. That's a permanent property, not a TODO — the first person to boot one is the validation.
Why no numbers?
We ran the full bench and the 8-pack in both reasoning modes. Then we checked per-card VRAM and found the dual had been running with pipeline parallelism silently disabled — it boots, serves, and passes every functional gate while degraded, and the only trace is one line in the boot log. Every number we had was taken on that path.
Rather than publish figures we'd have to retract, they're withheld until a re-run on the corrected config. The finding that caused it is portable, though, and is probably the most useful thing in this post:
The shipped default is now
-ts 0.55,0.45, which balances it. If you run any dual-card MTP config — this model or not — check for it:Nothing else surfaces it. Not health, not verify-full.
Things worth knowing before you launch
Single-card now reaches 262144 — by halving the KV to⚠️ This is a deliberate quality trade:
q4_0. The oldiq4nldefault (q8_0 KV) capped at 131072 (196608 failed oncompute pp buffers, 262144 onrs cache). Theiq4xsdefault swaps toq4_0KV, which halves the cache and clears both walls, so the full 262K fits on one card.q4_0KV is below our usual q8-grade serving floor (KLD ~5.75× worse than q8_0, and never depth-validated on this DeltaNet-hybrid family) — treat this variant as a max-context / vision exhibit, not the serving-grade tier. Want q8-grade KV back?KV_TYPE=q8_0 CTX_SIZE=131072restores the old config on the same slug. Context depth: NIAH-clean to 240,635 tokens (91% of n_ctx); the fine ladder past that is unrun.This model has two sampler rows, one per reasoning mode, and they're from the model card rather than our usual Qwen defaults:
presence_penaltymatters most — the card zeroes it for thinking because a reasoning trace legitimately repeats.xhigh, the maximum — but since 2026-08-16 the vLLM composes ship a server default oflowfor thinking boots, which injects a brevity instruction ("keep your thinking brief and focused, moving directly to the conclusion").lowis an active nudge, not a middle setting: the template injects a verbosity instruction forxhigh, a brevity one forlow, and nothing formedium— somediumis the un-nudged baseline, not the midpoint.xhigh/medium/lowONLY.highbehaves differently per engine: on vLLM it raises (Unexpected reasoning effort …); on llama.cpp the GGUF's embedded template silently remaps it toxhigh— the most expensive level, and the one that blows the timeouts below.SPEC=off.Vision: working across the vLLM dual slugs and the iq4xs single-card. Both checkpoints — int4 AutoRound and the official FP8 — ship the full vision tower (the FP8 does not skip it: 333
model.visual.*tensors), and image + video input is verified on the dual vLLM slug (2026-08-16 — colour/layout and a 3.25s clip described correctly; input-only → text out). The iq4xs single-card slug also mounts unsloth'smmproj-F16.gguf(928 MB,WITH_VISION=1) and served one correct high-res recognition. Grounding / OCR-accuracy runs still welcome.Getting it
Weights are public.
setup.shfetches them:Run it
--forceis required — 🐣 Incubating slugs are launch-gated and hidden fromswitch.sh --list(reveal with--list --all).All serve an OpenAI-compatible API as model
qwen3.8-27b.--owuiwires Open WebUI.Thinking mode is a boot-time property on llama.cpp and per-request on vLLM:
vLLM also accepts thinking per-request without a reboot — send
chat_template_kwargs: {"enable_thinking": true}plus the thinking sampler;--reasoning-parser qwen3is always loaded.Run the evals yourself (or add your rig to the matrix)
Two non-overlapping passes: behavioural quality (the 8-pack) and operational health.
One-time setup for the three sandboxed packs:
pip install git+https://github.com/noonghunna/benchlocal-cli.git git clone https://github.com/noonghunna/benchlocal-cli bash benchlocal-cli/tools/build-sandboxes.sh # ~30 GB free; prune if tightOperational
report.sh --fullis the one to run if you're contributing — it bundles verify + stress + soak + bench + agentic into a single pasteable block with host details redacted.Quality — both reasoning legs
Instruct is the shipped default on both engines, so leg A is the bare launch — no env vars.
Verified by rendering the composes:
offTHINKING=1onfalseENABLE_THINKING=truetrueafter a thinking run needs a reboot too — not just the thinking leg.
llama.cpp —
THINKING=1flips reasoning and the whole sampler (it appends the card's thinkingrow, and llama.cpp takes the last value of a repeated flag). An explicit
TEMP=still wins.vLLM —
ENABLE_THINKINGflips the template only; the sampler stays on the Instruct row, soset it yourself. Miss the three sampler vars and you run reasoning at
presence_penalty 1.5, whichthe card warns causes language mixing. (Tracked in #1014 — when that lands this collapses to one
variable like llama.cpp's.)
Any slug substitutes:
llamacpp/qwen38-27b-{single-iq4xs,dual-q8kxl}·vllm/qwen38-27b-{dual,multi4,multi8}-max.Why each flag — these aren't decoration; each one exists because its absence produced a wrong number for us:
--sampling-from-servertemperature=0. Without it you measure greedy decoding, not the shipped config.--max-tokens 4096token_limitfailures that look like wrong answers.--thinking-max-tokens 16384--timeout-per-case 600timeoutfailures that read as content misses.REASONING_EFFORT=reasoning_effort, so it is the CALLER's choice, not a server setting. Since #1029 the composes default tolowserver-side; pin it anyway so the run records what it measured.lowscore and anxhighscore are different measurements, so state which you used.highdiverges by engine — vLLM raises, llama.cpp silently remaps toxhigh. Onlyxhigh/medium/loware portable. The effort-vs-quality tradeoff is unmeasured on this model — that sweep is owed. (The instruct leg always sendsnone, so it is unaffected.)--repeat 3--sampling-from-serverboth legs are sampled, so single draws aren't quotable.timeoutandtoken_limitare harness artifacts; onlyverifier_fail/wrong_answerare the model. A score that looks catastrophic is often a budget that was too small.cli-40has a second, independent timeout — setBENCHLOCAL_MODEL_TURN_TIMEOUT=900on the thinking leg, don't wait to be bitten. The two govern different layers, which is why raising one does nothing for the other:--timeout-per-caseBENCHLOCAL_MODEL_TURN_TIMEOUTBecause the sandbox cap is flat, whether you hit it is arithmetic rather than luck: one 16,384-token thinking answer takes
16384 / decode_TPSseconds, so anything slower than ~55 tok/s busts it.llama.cpp slugs and single-card configs sit below that line. The failure reads as a
timeoutrow, which looks like the model gave up — it didn't.ℹ️ benchlocal-cli has a
--model-turn-timeoutflag, butquality-test.shdoes not forward it — the env var is the only route through the wrapper.--no-thinkingagainst a server booted with reasoning on gives you a second thinking leg wearing the wrong label, and the A/B reads as a clean null.What'd help
-ts 0.55,0.45split.multi4andmulti8have never started. If you have the cards, you'd be the first — and at 8 cards note that KV heads replicate rather than shard (4 KV heads, 8 ranks), so expect weight headroom rather than a KV win.retrying without pipeline parallelism. We'd like to know how widespread this is.Drop results in a comment or use the
numbers-from-your-rigtemplate.Credits
draft-mtpspeculative decoding) and vLLM (--speculative-configMTP).All reactions