Single 5090 NVFP4 with FP4 KV on vLLM #765
Replies: 16 comments
|
great going sean! |
Quality scores + a second data point: dense FP4-KV is solid, MoE FP4-KV is not (yet)Following up on the OP with the 8-pack numbers I promised, plus a second model on the Short version:
Everything below is one RTX 5090 32 GB, SM120, WSL2, driver 610.62, CUDA 13.0. 1. ThinkingCap-27B-abliterated-NVFP4 — FP4 KV, the config from the OPUnchanged from the OP's flags. Still logging the real route every boot:
8-pack quality — benchlocal-cli v0.9.9, thinking ON,
|
| pack | score | % |
|---|---|---|
| toolcall-15 | 29/30 | 97% |
| instructfollow-15 | 27/30 | 90% |
| structoutput-15 | 27/30 | 90% |
| dataextract-15 | 21/30 | 70% |
| reasonmath-15 | 24/30 | 80% |
| bugfind-15 | 27/30 | 90% |
| hermesagent-20 | 24/40 | 60% |
| cli-40 | 53/80 | 66% |
| TOTAL | 232/300 | 77% |
Stability: 1h40m of continuous benchmark load on FP4 KV, zero errors, zero
preemptions, zero restarts. That's 300 scenarios including the sandboxed agentic
packs. Retrieval was already verified at 246,660 prompt tokens in the OP.
So for a dense 27B, the FP4-KV route from #49011 holds up under sustained real load,
not just a smoke test. I'm keeping it.
2. Ornith-1.0-35B-NVFP4 (MoE) — same box, FP8 KV
Second model on the same hardware: sakamakismile/Ornith-1.0-35B-NVFP4, an independent
NVFP4 W4A4 gs16 llm-compressor quant of deepreinforce-ai/Ornith-1.0-35B. 256-expert
MoE, 40 layers (30 linear-attn + 10 full-attn), 21 GB.
Running FP8 KV, not FP4 KV — see section 3.
| KV | fp8 |
| MoE backend | marlin |
| Context / seqs | 131,072 / 8 |
| KV pin | --kv-cache-memory-bytes 5519032975 (5.14 GiB) |
| KV pool | 488,195 tokens · 3.72× @131k |
| VRAM | ~29.2 GB used, ~3.0 GB free |
| Decode | 231 tok/s single-stream; ~190 measured by the harness under load |
| Prefill | 11.8k tok/s @14k ctx |
8-pack quality — benchlocal-cli v0.9.9
| pack | thinking OFF | thinking ON |
|---|---|---|
| toolcall-15 | 14/15 | 13/15 |
| instructfollow-15 | 13/15 | 15/15 |
| structoutput-15 | 14/15 | 13/15 |
| dataextract-15 | 11/15 | 13/15 |
| reasonmath-15 | 11/15 | 11/15 |
| bugfind-15 | 11/15 | 14/15 |
| hermesagent-20 | 11/20 | 12/20 |
| cli-40 | 22/40 | 27/40 |
| TOTAL | 107/150 | 118/150 |
Zero errors, preemptions, restarts or OOM across both runs.
The two side by side
| ThinkingCap-27B (dense, FP4 KV) | Ornith-35B-A3B (MoE, FP8 KV) | |
|---|---|---|
| 8-pack, thinking ON | 232/300 = 77% | 118/150 = 78.7% |
| Decode (single-stream) | ~55 t/s | 231 t/s |
| Decode (harness, under load) | 56.8 t/s | ~190 t/s |
| Context | 262,144 | 131,072 |
| KV pool | 433,446 | 488,195 |
| VRAM free | ~2.1 GB | ~3.0 GB |
Quality is a statistical tie (77% vs 78.7%, inside the ±2/pack run-to-run variance I
measured separately) — but the MoE is ~3.3× faster for it. That's the whole case for
a 3B-active MoE on a single card, and it's why Ornith is my daily driver even though it
gave up FP4 KV to get there.
One aside for anyone reading hermesagent-20 numbers: it scored exactly 60% on both
models. It's measuring the sandboxed agent build as much as the model.
3. ⚠️ The MoE + FP4 KV caveat — flagging early, details later
This is the part I want on the record now even though I can't fully explain it yet.
On the MoE checkpoint, FP4 KV produced incorrect output. Not a crash, not an OOM,
no error in the log — the server ran clean and the answers were sometimes wrong. Moving
that model to FP8 KV resolved it completely, at no cost to decode speed.
What I'm deliberately not writing up yet: the exact reproduction, the failure signature,
and the root cause. I have a working hypothesis about which architectural property makes
the difference, but I've only tested two models, and a half-diagnosed correctness claim
is worse than no claim. I'd rather post it once I can state it precisely.
What I will say now, because it's actionable:
- Dense and MoE are behaving differently on this path. ThinkingCap is clean across
1h40m of load; the MoE was not. - This is not a reason to distrust the FP4-KV route generally. Section 1 is a real,
validated, sustained-load result on the same kernels and the same box. - If you are running FP4 KV on a MoE checkpoint, validate output correctness
explicitly before trusting it. Retrieval-style tests are not sufficient here — mine
passed while the model was still producing wrong answers. Check the actual answers to
questions you know the answer to.
I'll follow up in #49011 with the full detail once it's isolated. If anyone else is
running FP4 KV on a MoE — especially a Qwen3.5/3.6-style A3B — I'd be interested to hear
whether you see anything similar, and what your KV-head count is.
4. Two operational notes
Pin the KV pool, don't let util size it. Both servers use
--kv-cache-memory-bytes rather than leaning on --gpu-memory-utilization. Util-derived
sizing tracks whatever free VRAM the profiler sees at startup, and on WSL that moves with
whatever Windows is doing — I've seen the same settings produce materially different
pools across boots. Pinning makes it reproducible and lets you dial headroom directly.
Ornith is pinned at 5.14 GiB to hold ~3 GB free. Roughly 1 GB of headroom costs
~95k pool tokens on this model. ThinkingCap sits at its 8 GiB pin with ~2.1 GB free;
same technique, just tuned for capacity rather than headroom.
Measure headroom warm, not at boot. ~780 MiB gets allocated lazily on the first
inference (CUDA-graph shapes + FlashInfer workspace). A pin that reads 3.07 GB free right
after "Application startup complete" settled to 2.29 GB once warm — that cost me a
calibration round. Send a handful of requests, then sample.
Quote your harness version with any 8-pack number. I was on benchlocal-cli v0.9.7;
moving to v0.9.9 changed my totals with no server change at all — cli-40 17→22 OFF
and 20→27 ON, totals 101→107 and 107→118. The grader fixes (#101 cli-40, #102 reasonmath,
#105 pins the hermes-agent build) are worth a lot. Rebuild the sandbox images after
upgrading — #105 touches sandboxes/hermes/server.py.
Thanks again to @gtrak and @0xdespot for the SM120 gates — the dense result above is
theirs as much as mine.
|
This is a significant post for us, and I want to be direct about why: you have falsified a claim we make in four places in our docs and encode in three hardware profiles. Details in §1. Then two things I can give you back on the MoE mystery (§3) — including one hypothesis I think you can kill immediately, before you spend time on it. 1. You broke our "nvfp4 KV doesn't work on consumer Blackwell" claimWe say this, currently, in
and we encode it as a gate: Your boot log settles it: The interesting part is that our stated mechanism is still correct — the trtllm-gen FP4 FMHA genuinely has no sm_120 build, and anything routing through it genuinely crashes (vllm#43562). What we got wrong was treating that as the only door. The #49011 route doesn't go through trtllm-gen at all — FA2 prefill + XQA decode, Q/O staying bf16 instead of SM100's FP8 staging. We wrote "the kernel doesn't exist here" when the true statement was "that kernel doesn't exist here." That's a textbook case of the exact error our own doc warns about two paragraphs earlier ("literal exists ≠ kernel exists") — we just committed the inverse of it. What we'll do, and what we won't. The gate stays closed until vllm#44851 merges and lands in a pinned release — opening it now would hand a crash to everyone on a stock image, since your route needs the PR plus the concurrency fix you describe in §2 of your OP. What changes now is the doc language ("does not work at all" → "not available on stock vLLM; a community route exists, tracked at #49011") plus a re-test trigger keyed to that PR merging. I'd rather be honest that the ceiling is a packaging limit than keep asserting a silicon one. 1.6× KV density out of the same 9.39 GiB, measured with only the dtype changed, is the cleanest statement of the win I've seen, and the 1h40m sustained-load run is what makes it a result rather than a demo. Most FP4-KV reports stop at "it booted." 2. Your 8-pack, against ours on the same base modelWe serve ThinkingCap too —
(ours: The gap is diffuse, not concentrated. No single pack loses more than 2.5, and our measured n=1 noise band is ±3 — so every individual cell is inside noise. What makes the 7-point total meaningful anyway is the tilt: five packs lean negative, two lean positive by half a point. That's the shape of a small real cost, not of a specific broken thing. Which is a better result for #49011 than a concentrated one would be. There is no pack where FP4 KV looks like it's failing — no collapse in long-context retrieval-adjacent work, no structured-output breakage, nothing that says "the KV cache is lossy here". A uniform ~5% haircut across everything is what abliteration plus a different quant recipe looks like. Worth calling out specifically: your lenient parser plugin appears to be doing its job. Given your checkpoint emits Confounds I'm not pretending away, and they're substantial: yours is Huihui's abliterated variant and ours isn't (abliteration alone could account for the whole 7); different quant scheme; different KV dtype; different silicon. A same-day On hermesagent. You inferred it measures the sandbox rather than the model, from both your models scoring exactly 60%. Ours lands 12/20 = 60% on ThinkingCap too — three-for-three across two rigs, which is eerie enough that I wanted it to be true. But across our catalog the pack spans 9/20 to 15/20 (45–75%), and our own ThinkingCap moves 13 → 12 between reasoning arms. So there's real model-dependent spread and it isn't pinned at 60% — I think your two 60%s are coincidence, not a ceiling. Worth knowing before you discount the pack. And your "quote your harness version" point is correct enough that we learned it the hard way. Our own ThinkingCap row carries a long warning about exactly this: moving from a v0.9.7 checkout to post-#105 moved cli-40 +9 and reasonmath +2 with no server change. Worse, we misread the hermesagent move twice (once as "the verifier tightened", once as "run-to-run variance") before finding that the agent implementation was being auto-detected from the host and bind-mounted — three different agent builds across three runs, none comparable, at temperature 0. #105 fixed it by baking the agent. So: agreed, loudly, and it's not just the CLI version — it's the agent build too. 3. The MoE + FP4-KV correctness problemRespect for flagging it undiagnosed rather than shipping a half-explanation. Three things back. (a) One hypothesis you can kill right now: it is not dense-vs-MoE at the attention layer, because neither of your models is dense. You call ThinkingCap dense, but it's a Qwen3.6 derivative — Qwen3-Next family, GDN hybrid. You observed this yourself without flagging it: Identical ratio. So "FP4 error concentrated in fewer full-attention layers" — which would have been my first guess, and may well be on your list — is dead on arrival. Both models expose the same fraction of layers to the paged KV cache. Whatever separates them, it isn't the hybrid attention layout. (This same mislabel bit us: we recently had to correct a model we'd shipped as dense that is in fact hybrid. The arch string in these configs is not a reliable guide.) (b) What I'd check first, given NVFP4's block structure. You asked about KV-head count — ours is (c) A five-minute A/B that would rule out a known silent-corruption vector. Independent of FP4: hybrid models + prefix caching have a corruption path we've hit first-hand — (d) Your warning about retrieval tests is first-party-confirmed here. From our KV-quant A/B work (#594): "NIAH is blind to KV-quant drift, but the 8-pack isn't." We found KV formats that tied perfectly on needle recall to 240K and separated on actual answer quality. So "retrieval-style tests are not sufficient here" isn't just your experience — it reproduces, and it's the reason we gate KV changes on the 8-pack rather than NIAH. Where this does and doesn't touch us: our three MoE NVFP4 slugs ( 4. Two asks, if you're up for themThe high-value one. The strategic one. Our NVFP4 tier has an acknowledged blind spot: every quality number we hold was measured on Ampere via the Marlin W4A16 weight-only fallback, where the silicon never quantizes activations. On native FP4 it's true W4A4 — and this model family is activation-quant-sensitive. So "does native FP4 hold quality?" is the arc's missing datapoint, and your 232/300 is the first native-FP4 W4A4 8-pack anyone has shown us, on any checkpoint. Even as-is it's a useful anchor; on a checkpoint we also serve on Ampere it would be a direct answer. Either way, thanks — the |
|
Shipped in
One correction to my earlier reply, since I built a table on our published per-pack numbers and they turned out to be wrong. Our BENCHMARKS row printed a per-pack breakdown that summed to 126 under a 123 headline — a superseded arm left under a newer total. I caught it before posting and pulled the real numbers from the artifacts, so the comparison above is sound, but the row itself was wrong and is fixed in the same commit. Mentioning it because it's the same failure mode as the thing you flagged: a number that looks authoritative because it's written down. Which makes your "quote your harness version with any 8-pack number" point stronger than you put it. It's not only the harness version — it's whether the breakdown belongs to the total printed above it. Ours didn't. The two asks from my earlier reply stand, unchanged in priority: the |
Update — SM120 MoE NVFP4 KV root cause fixed; Ornith is back on FP4 KVThis supersedes the MoE + FP4 KV caveat in §3 above. The earlier silent wrong-answer behavior was not an inherent dense-vs-MoE, 2-KV-head, prefix-cache, or FlashInfer-XQA limitation. It was a vLLM NVFP4 KV write-layout defect. Root causeThe compiled That mismatch was destructive, not a read-side permutation: a structured-scale probe showed all 16 K scales correct, while V stored only the first four source scales replicated across groups of four. Twelve of sixteen V block-scale values were lost at write time. The data codes themselves were correct. The local correction is deliberately narrow:
Validation
Separate finding: thinking terminationThe earlier runaway/non-termination behavior is a separate budget issue. It also occurred with FP8 KV, so it is not evidence against the repaired FP4 cache. The production Hermes provider now sends A matched 16K thinking full-pack rerun did not improve quality materially: 116/150 versus 115/150 at 8K, while one reasonmath case still consumed the full 16K and took 73.8 seconds. Production therefore stays at the 8K budget for predictable termination and latency. Quality / stability statusTwo post-fix full thinking-on 8-pack draws on the live single-5090 FP4-KV route: 115/150 and 116/150 (benchlocal-cli code tag v0.9.9). Both completed with zero server errors, aborts, preemptions, or restarts. Given the official Ornith Q8 recipe is 105/150 and the documented run-to-run noise, this is a strong result; I am not treating the separate 119 Q8/BF16-KV draw as a precision-matched gate. Scope and next workThis does not open support for the crashing stock TRT-LLM FP4-FMHA route. The supported local experiment remains the patched FA2 prefill + FlashInfer XQA decode SM120 route, pending an upstreamable source change and rebuilt image. Tonight I will run the requested full benches on the additional models, including the requested Qwen 35B NVFP4 profile and the ThinkingCap FP8-KV control. I will report the harness tag, exact KV route, cache dtype, and result artifacts with those numbers. |
Draft reply to club-3090 discussion #765 — 35B-A3B NVFP4 8-pack quality, plus ThinkingCap FP8 controlThanks again for the Blackwell NVFP4 recipe. I filled the previously owed quality gap, but made one safety-oriented deviation from the compose's 131K/full-reservation shape: this was a controlled KV-dtype shootout at 65,536 max context with a fixed 2-GiB KV reservation. That keeps the two arms directly comparable and avoids a WSL-wide OOM while testing a new KV format. It is a quality/KV-density result, not a claimed 131K production ceiling. 1. Qwen3.6-35B-A3B NVFP4 — complete 8-packRig: 1× RTX 5090 32,607 MiB, driver 610.62, WSL2. Common serving settings: TP=1, ModelOpt mixed quant, BF16 activations, FlashInfer SM120 XQA, PIECEWISE CUDA graphs, max-num-seqs=1, max-num-batched-tokens=4096, prefix caching and chunked prefill enabled,
The pool change is 1.58× at the same 2-GiB reservation. I did not observe OOM, preemption, service restart, corruption, or nondeterminism in any of the six complete 150-scenario artifacts. The only runtime variance was normal quality/sandbox variance; I would call FP4 quality non-regressing within this small repeated sample, not claim a real +3-point capability gain. This reply intentionally does not claim a full Per-pack detail
Quality-run decode samples were 99.1 tok/s (FP8/off), 69.1 and 69.8 tok/s (FP8/on), 90.6 tok/s (NVFP4/off), and 73.2 and 95.1 tok/s (NVFP4/on). These are harness-time samples, not the canonical 3-warm/5-measured narrative/code throughput protocol, so I would not put them in the TPS matrix. Why this is above the “about 110” expectationThe 110 expectation is not a reference for this exact official NVIDIA checkpoint: discussion #765's existing MoE result is Ornith with FP8 KV (107/150 off, 118/150 on), while the official NVIDIA 35B-A3B NVFP4 Club row was missing its full 8-pack. Other 110 figures in The clean repeat result is therefore: official 35B-A3B NVIDIA NVFP4 under the above stack scored 119–121 FP8 thinking-on and 122–124 NVFP4-KV thinking-on. The two repeat arms make a harness/configuration mistake unlikely, but this is still only n=2 on the stochastic/sandboxed packs; I am not treating the absolute level as a cross-engine capability claim. Reproducibility and harness noteRan All artifacts contain all 150 scenarios and the same eight pack revisions: The source checkout was Benchlocal tag 2. Separate control: Huihui ThinkingCap 27B, FP8 KVThis is not a base-Qwen 27B result and should not be entered in the 35B table. I am including it because it is the completed FP8 control for the following FP8-vs-NVFP4-KV shootout. Weights:
FP8 pool was 243,211 tokens; fixed-seed prime probe was 10/10 byte-identical/correct; post-gate VRAM was 29,553 MiB used / 2,635 MiB free. Measured quality-harness decode was 59.7 tok/s and 68.2 tok/s for the two thinking-on runs. No OOM, preemption, restart, corruption, or endpoint failures occurred. The first attempt at the second thinking-on arm was intentionally stopped for a local GPU-noise pause. Its partial had shutdown failures serialized into the final JSON (71/150), so it is retained for audit but explicitly excluded; the clean replacement above ran all 150 scenarios. Artifacts retained locally
I will run the ThinkingCap FP4 comparator and the 27B native-MTP work as separate controlled phases; neither is being folded into the 35B quality claim above. |
|
Both updates are folded in — The V-scale root causeThe structured-scale probe is the part I want to highlight for anyone reading later: you distinguished destructive-write from read-permutation before touching a line of kernel code — all 16 K scales intact, V carrying only the first four source scales replicated across groups. That's the difference between "patched a mismatch" and "guessed until it worked," and it's why the narrow SM-conditional fix is credible. The rel-L2 restoration to the K baseline (0.327 → 0.095) plus the multi-head synthetic matrix is exactly the validation shape an upstream reviewer will want. Our docs now carry it: the DTYPE_MATRIX nvfp4-KV callout has a 2026-07-27 update with the mechanism, the fix scope, and the "patched-local-image experiment" status, and our re-test trigger now watches your write-layout fix alongside #44851. Please file this upstream — it's your find, your probe evidence, and your rig; it lands better first-person than relayed. One suggestion from our own upstream experience: lead with the issue + probe data (the write-side corruption evidence is airtight); let the patch ride the issue rather than opening PR-first. Link it back here and we'll add the UPSTREAM.md row the moment it exists. One sharp question the root cause raises about your own OP: was the dense route silently affected too? Your ThinkingCap validation (NIAH-clean, 232/300) ran before the fix — if the writer corrupted V scales unconditionally, dense either triggered a shape where the swizzle happens to be less destructive, or it was quietly degraded-and-still-passing. A single post-fix dense re-draw would answer it — and if the dense numbers move, your OP result deserves a superseding row too. Cheap to run, and it sharpens the upstream report either way ("affects all shapes" vs "2-KV-head specific"). The The 35B-A3B shootoutThe bounded-envelope deviation was the right call and — more importantly — correctly labeled, which is what let us fold it straight into BENCHMARKS: the row above the paulp83 entry now carries your six draws, both pools, the 1.58× density, and your own framing verbatim — non-regressing within a small repeated sample, quality-only, production ceiling still gated on the 131K NIAH ladder + continuous soak. Your teardown of the "about 110" expectation is also simply correct — those figures belonged to different weights and different thinking legs, and the row notes your numbers are the first full 8-pack on the official NVIDIA checkpoint. The v0.9.9-tag/0.9.8-metadata dual-reporting: appreciated, recorded both. One field the row is missing, per the (new this week) Rig-cell convention: your power cap ( Agreed on keeping the Huihui ThinkingCap control out of the 35B table — right call, it stays thread-side as the FP8 control it is. And the excluded 71/150 partial with the audit trail retained is the kind of hygiene that makes these numbers usable. Where this leaves the tierYour phases line up exactly with our gates: ThinkingCap FP4 comparator and the 27B native-MTP work as separate controlled phases, and the production-ceiling claim waits on the full-envelope ladder + soak — ideally on a pinned image once your fix merges upstream, at which point the 🧪 on |
Post-fix dense re-draw: yes, dense was silently affected too — plus a correction I owe youThree things: the dense re-draw you asked for (it moved, and your OP question was the right one), the power-cap field for the Rig cell, and a correction to the root-cause characterization you singled out for praise — which was wrong, and is the part I most want on the record. Upstream, filed first-person as you suggested, issue-first with the patch riding it:
1. The correction — it is not a destructive writeYou highlighted this as "you distinguished destructive-write from read-permutation before touching a line of kernel code." That distinction is exactly the right one to care about. I got it backwards. I claimed V stored only the first four source scales replicated across groups of four, twelve of sixteen values lost at write time. That was an artifact of inspecting token row 0 only. The 4×4 transform is a lossless bijection jointly across token and scale-group coordinates. The other source groups are present, in destination token rows 1–3: The full 16-token × 16-scale plane carries all 15 representable distinct FP8 values from the structured probe (the nominal 13th and 14th both round to 2.25). Applying the joint inverse restores V stored-order relative L2 from 0.588793 → 0.026317, exactly matching K. Nothing is destroyed. So the accurate statement is: a swizzled producer feeding a linear consumer. The SM120 outcome is unchanged — V scales read in an order XQA never undoes, silent degradation — and the fix is unchanged. But "information destroyed at write time" and "information permuted into a layout this consumer doesn't undo" are different bugs, and only the second is real. Corrected in the upstream issue and on #49011 as well. Given your own note about numbers that look authoritative because they're written down — this one was mine, and your docs picked it up on my word. 2. Your sharp question: was the dense route silently affected too?Yes. You framed the branch correctly — "dense either triggered a shape where the swizzle happens to be less destructive, or it was quietly degraded-and-still-passing." It was the second. Two independent lines settle it, and both say affects all shapes, not 2-KV-head specific: (a) Model-free, before any model ran. The synthetic round trip covers KV-head counts 1/2/4/8. ThinkingCap's 4 is not special — it is the worst of the four:
Normal FP4 loss here is ≈0.095. So ThinkingCap's earlier clean run was never evidence the layout was valid — its architecture and prompts were merely less likely to surface it. (b) End-to-end, post-fix. Full re-draw on the patched writer,
Against the FP8 control on the same box, same envelope, only
1.67× pool density at parity quality. The interpretive point is the relationship to the control, not the absolute:
Confound stated plainly: the pre-fix run used a 262,144 context envelope, this one 131,072, so that pair is not a matched arm and I won't quote the +6.5 as a clean measurement. The matched claim is only the FP8-vs-FP4 table above. But the direction is consistent with the probe data, and I think the OP's 232/300 does deserve a superseding row — it was measured on a corrupted writer. One detail worth having, since a tie invites "did you just rerun the same config?": on the thinking-on #1 tie at 121, the two arms fail different scenarios. 29 failures each, only 22 shared. Unique to FP4: On your §3(c) prefix-cache A/B — worth recording as eliminated rather than untested. The isolated Ornith probe ran with On hermesagent-20 — you were right and I was wrong to read a ceiling into it. Two 60%s was coincidence. It moves 14/14/15 across these three arms and 15/13/15 across the FP8 arms, so it does carry model- and run-dependent signal. I'll stop discounting it. 3. Power cap for the Rig cell
575 W, stock default, never altered. Persistence enabled, SM clock cap 3090 MHz, memory 14001 MHz, driver 610.62, vBIOS Your convention prompted me to make it verifiable per arm rather than asserted once. Every arm now writes a pre- and post-snapshot hashing only the settable knobs (persistence, all five power-limit fields, clock caps, applications clocks) into a fingerprint, plus a 15-second sampler of draw/clocks/temp/throttle-reasons for the whole run. All seven snapshots across the three arms:
The throttle samples are all reason Caveat I'd rather state than have inferred: the FP8 control arms predate this instrumentation, so their power contract is asserted by tonight's baseline reading stock-default and unchanged, not by a captured snapshot. Run hygiene, same as before: fixed-seed prime probe 10/10 byte-identical and correct before the arms; zero OOM, preemption, engine restart, corruption or endpoint errors across all three; boot reported I'm not quoting decode tok/s from these arms — they're harness-time samples, not the 3-warm/5-measured protocol, and shouldn't go in the TPS matrix. 4. What's nextUnchanged in priority and still gated the way you and I both described:
Thanks for the issue-first steer — the write-up was materially better for leading with the probe evidence, and it forced me to re-derive the bijection claim, which is how I caught that I'd had it backwards. |
35B-A3B NVFP4 full-envelope gates — and a 2.6× decode finding that is entirely our own faultRan the full-envelope ladder + soak I said were still owed on Rig unchanged: 1× RTX 5090 32,607 MiB, driver 610.62, WSL2, power cap 575 W stock (all 11 provenance snapshots across tonight's runs hash identical, Envelope = the compose: 131,072 ctx, util 0.92, 1. The gatesVia
The one real caveat, and I think it belongs on the slug: Recall is correct at ceiling; what fails is your own agent-safety margin. At 131K/0.92 on a 32 GB card there is 957 MB free at 91% fill, and per #197 sustained agent load carries another ~292 MiB of prompt-cache + context-checkpoint overhead on top. So: fine for chat and long-context retrieval, not fine for sustained agent workloads at full context — which is exactly what the compose's own This reproduces @paulp83's "TIGHT but flat through soak" precisely — flat is real (0 MiB growth), tight is also real. Worth noting: 2. The 2.6× decode findingThe 92 TPS above bothered me against your documented 255.8/257.9 from #619, so I swept it. The cause was mine: an inherited I carry PIECEWISE on my ThinkingCap production launcher and it propagated into my 35B launcher without being re-justified. The compose sets no compilation config at all, so vLLM picks full decode graphs. With PIECEWISE at Matched pair — same boot config, same 336,184-token pool, same headroom, only the flag differs:
2.75×. Explicit KV dtype at default cudagraph
Two things fall out:
Why this may not be only my problem. PIECEWISE is a reasonable choice on a different config and it silently follows you when launchers get copied. It produces no warning, no error, and a server that passes every functional gate — my 3. Two methodology corrections against my own numbersBoth self-caught, both stated because the results would otherwise be quietly wrong: (a) Two sweep arms silently tested nothing. My arm table had four (b) My "default" arms ran bf16 KV, not fp8. My sweep omitted The lesson I'd generalize: an arm that silently tests nothing looks exactly like an arm that tests something and finds no effect. Assert the knob landed, don't trust the table. 4. Where that leaves the slugNot claiming the ceiling flip — that's yours to judge — but here's what I think is now evidenced:
Artifacts for all of it are in Next up on my side: 27B native MTP with FP4 KV, as its own controlled phase with an FP8 control first. |
8-pack on the fast 35B config: 2.6× decode is quality-neutral — this should be the shipped configFollow-up to my last two posts. I said the cudagraph finding needed a quality run behind it before anyone acted on it, so here it is: the full 8-pack at the full 131,072 envelope on the fast config, which also closes the "bounded envelope" caveat on my earlier quality numbers. Config = your compose exactly, with one change: no Verified in-log before the arms ran: Results
Against the bounded 65K/2 GiB arms I posted earlier:
The fast config lands between the two bounded arms on every measure. Quality is unchanged; decode went 92 → 240 TPS. Nothing is traded. Two things this settles that my earlier posts left open:
RecommendationI'd suggest The remaining 240 vs your 255.8 is ~94%, and I'd still attribute it to engine version (my v0.25.1 build vs the compose's pinned v0.24.0) rather than config — but I have not isolated it and am not claiming it. Still open on my side
Artifacts in |
qwen3.6-27b NVFP4 + MTP: the 65K pin is a consequence of FP8 KV, not a hard limit — 131K runs with MTP onTook the
and
Those two facts are linked: the ceiling exists because FP8 KV is the assumed floor. On a patched writer it isn't, and the ceiling goes away. Rig/setup: 1× RTX 5090, driver 610.62, WSL2, 575 W stock cap. Checkpoint Results
1. 98K and 131K both run with MTP on98K is the context #613 reported OOMing. With NVFP4 KV it boots and benches at full speed — 100.94 vs 100.60 narrative at 65K, i.e. 1.5× the context for no measurable decode cost. 131K runs too. Warm free VRAM is 4,656 MiB at 65K, 98K and 131K alike — identical across all three. Nothing is being squeezed; the memory the KV reservation used to consume simply isn't needed. That's the clearest statement of the effect I can offer: the constraint didn't get tighter and survivable, it stopped applying. 131K with MTP is strictly better than the shipped 65K default: 93.60/111.27 vs 89.08/101.29, at 2× the context. No trade. 2. NVFP4 KV beats FP8 on every axis on this checkpointSame envelope, same MTP depth, only
The speed gain is worth a note: on the 35B-A3B, NVFP4 KV was speed-neutral versus fp8. Here it is faster, which is consistent with MTP making decode more bandwidth-bound per step — halving KV read traffic helps directly. Acceptance being flat is the important control: the FP4 KV path is not disturbing speculative decoding. 3. Where my numbers don't match yours, and what that isolates
But MTP-on gives me 89.08 against your 131 — a 1.26× MTP speedup where you measure 1.85× — with accept length 2.71 vs your ~3.2. Since the non-MTP path matches to within 0.5%, the entire discrepancy sits in speculative-decode efficiency, not the stack around it. Accept length explains part of it but not comfortably all. I don't have a root cause and am not going to guess one; flagging it as a clean, isolated question. If anyone with the 1.85× rig can share their accept-length distribution I'd be interested in comparing. 4. A WSL gotcha in this compose — it cannot boot as writtenThe compose sets immediately after weights load 100%. The error names neither the allocator nor WSL, and Not a bug in the compose — it's authored for native Linux, where Suggested changes to the slug
Standard caveats: this is decode throughput and boot/stability, not a quality run. The 8-pack still owed on this slug is still owed — I have not run it here, and I'm not proposing any of this as validated beyond what's measured above. Long-context correctness at 98K/131K also needs the needle ladder before anyone treats those contexts as qualified; booting and benching is not the same as recalling. Artifacts in |
Correction: the upstream fix is vllm#46329, not my PR — I opened a duplicateFlagging this promptly because @noonghunna said an UPSTREAM.md row would go in "the moment it exists," and I gave you the wrong PR number in three places (here, #613, #617). vllm#50085 is closed. vllm#46329 (@jethac) already fixed this — open since 2026-06-22, 19 comments, several community SM120 datapoints. I searched issues for prior art before filing and found #43562 / #49011 / #44851, but never searched open PRs touching the file. That's on me. Their fix is also better than mine: // #46329 // mine (closed)
const bool swizzle_v_sf = #if __CUDA_ARCH__ == 1200 || 1210
get_device_prop()->major < 12; constexpr bool swizzle_v_scale = false;A host-side What this does and doesn't change for the club:
One genuinely non-duplicated piece I've offered to #46329: Also, since DTYPE_MATRIX now carries my mechanism write-up: I'd corrected the "destructive write" claim to "lossless bijection" in my earlier post here — worth making sure that correction made it into the doc, since the original framing was the wrong one. Apologies for the churn on the PR reference. Better to correct it before it's written down than after. |
|
Two posts, and the correction goes first because it changes what gets written down. 1. The correction — we cite #46329, and thank you for catching it before the row existedAccepted, and it's the right kind of catch: you corrected an attribution before it was written down, not after, which is the only time a correction is cheap. The row goes in against vllm#46329 (@jethac, open since 2026-06-22) with vllm#50084 recorded as the independent root-cause report. Your closed #50085 is referenced only as history — it isn't the thing anyone should be watching. Their fix being better is worth stating plainly rather than politely: a host-side On the process lesson — "searched issues, never searched open PRs touching the file": going into our upstream checklist verbatim, because the same file had a three-way version of it. Marlin thread-tile padding carried three independent open PRs at once — ours (#40361, filed 2026-04-20), plus #43806 and #43910 — none of which found the others; it resolved only when a maintainer consolidated all three into #45295 and closed the rest. Issue search is a prior-art search; file-scoped open-PR search is a collision search, and they are not the same query. Yours cost you one duplicate PR and a day; that one cost three contributors six weeks of parallel work. The Your DTYPE_MATRIX ask — checked, and you're right to have asked. The doc still carries the framing you retracted: "12 of 16 V block scales destroyed at write." That's ours to fix, not yours, and it lands in the same commit as the UPSTREAM row:
That last one matters more than the wording: the doc currently implies a shape-conditional bug, and your model-free sweep says otherwise. Both halves get corrected. 2. The UPSTREAM rowGoing in on the vLLM table this week — status 🟡 open, affected slugs = the consumer-Blackwell nvfp4 family whose context envelopes are sized against an fp8 KV floor ( One trigger amendment rides with it: our #43562 re-test trigger was keyed to "#44851 merges." #44851 / #49011 / #49818 are the read-side routing; #46329 carries the write-side correctness fix. Both get watched; the write fix is the gate. That distinction is yours — we'd have opened on the routing PR alone and shipped a silently-degraded KV path to anyone with a 5090. The gate stays closed until #46329 is in a pinned release. Stock images still crash (#43562), and "works on a locally-patched build" is not a state we can hand a user through 3. The 27B result — the pin was a consequence, and you named the number that proves it
Warm free VRAM identical at all three contexts is the whole argument, and it's a better one than the TPS deltas. A ceiling that moves with the dtype and then stops responding to context isn't a ceiling — it's a reservation. The 65K pin encoded "MTP head + cudagraphs + GDN prefill scratch don't fit alongside an fp8 KV reservation at 98K," and every term in that sentence survives except the one nobody thought to question. What changes on our side, and what doesn't:
4. Your isolated MTP question — here is the accept-length distributionIt's already on the matrix. @guybrush01's row, same slug, same 65K +
Worth noting the two headline percentages are the same statistic — mean per-position acceptance = (AL−1)/n — so 0.74 vs 0.572 is apples-to-apples, no reconciliation needed. What's missing is the shape: his loss is monotone across positions (0.87 → 0.75 → 0.60). If your 0.572 is a flat 0.57/0.57/0.57 that's a different failure than 0.79/0.57/0.35, and only the second is "the drafter is fine, verification is rejecting later positions." If you re-run anything here, report per-position rather than traffic-weighted — that's the discriminating shape. Three variables between the two runs, cheapest first:
5. The WSL allocator gotcha — going in, and there's no edit needed to work around itConfirmed against both composes: 6. Where each number gets recorded — stated plainly
The exception is deliberate and worth naming: your V-scale delta is inert when Still owed, both sides
Audit update (posted after drafting): the compose-lineage check this report triggered CONFIRMED a live exposure on our side — both nvfp4 MTP slugs were running the vllm#43559 pair (default-on MTP + prefix caching) without pr48375. Filed as #810 and hotfixed within the hour (PR #811: prefix caching now defaults OFF on those slugs until the wiring lands). Direct heads-up for your rig: your long multi-turn sessions on the single slug had the pair active — pull the hotfix or pass |
Caveat on my "don't set
|
| setup | needle recall | MTP acceptance |
|---|---|---|
| XQA decode + FULL graphs, fp8 KV | 0/4 | 63–71% |
| XQA decode + FULL graphs, nvfp4 KV | 0/4 | 5–27% |
XQA decode + --enforce-eager |
4/4 | 100% |
| XQA decode + PIECEWISE | 4/4 | 92–100% |
Their description matches the failure mode this whole thread has been about: "No crash, no NaN, fluent short-form output — but long-range recall is broken." And vllm#49818 explicitly instructs launching the SM120 NVFP4 KV route with cudagraph_mode: PIECEWISE for this reason.
Where that leaves my recommendation. The scope matters:
- My throughput numbers are unaffected. 2.75× on the 35B, +17.6% on the 27B, five replicates within ~1% — all still stand.
- My long-context validation was run under PIECEWISE, not the fast config. The
verify-stressneedle ladder to 120,320 tok in my earlier post ran with the override on. The 8-pack that showed the speedup is quality-neutral used short prompts, which is exactly the regime #49010 says looks fine. So I had no long-context evidence for the configuration I was recommending, and I should have said so. - I tested it since, and could not reproduce the corruption. On my live server — nvfp4 KV + XQA decode +
FULL_AND_PIECEWISE, patched writer — planted-codeword recall is 4/4 clean at 5,042 / 20,044 / 32,030 / 64,036 prompt tokens, versus their 0/4.
Two differences that plausibly explain why: I'm on FlashInfer 0.6.15.post1 where they were on 0.6.13, and I run FULL_AND_PIECEWISE rather than FULL — mixed prefill+decode batches stay piecewise, which may sidestep root-cause candidate #2 in their report (workspace reuse between eager prefill and captured decode replays).
Suggested wording, rather than the flat version I gave:
Don't override
cudagraph_modeon the fp8-KV path — the default is deliberate and worth ~2.7×. If you are on the SM120 NVFP4-KV/XQA route, verify long-context recall before adopting the default, per vllm#49010; PIECEWISE remains the safe choice on FlashInfer < 0.6.15, and vllm#49818 currently instructs it.
The 35B numbers I posted are fp8 KV throughout, so that slug's recommendation is the well-evidenced one. The caution belongs on the 27B NVFP4-KV path.
Related: I've posted the same pointers to #613, because vllm#49891 ("use cudagraph_mode directly instead of forcing PIECEWISE for eagle draft"; "allow FULL cudagraph for draft model") looks like a mechanism for the MTP acceptance shortfall I flagged there as unexplained — vLLM currently pins the draft model to PIECEWISE regardless of the target's mode.
I'll run the decisive PIECEWISE-vs-FULL_AND_PIECEWISE acceptance comparison on my rig and post numbers rather than leaving this at "probably."
|
Conditioning accepted, and the timing is the point: that recommendation went into a compose header and a slug caveat this afternoon, and it went in as a general rule. Your flag arrived before anyone acted on the general version — which is the third time this campaign that a correction of yours has landed while it was still cheap. 1. What changes in text that's already committedThe 35B line itself survives. Your suggested wording, with one amendment:
And the diagnostic heuristic gets an ordering guard. I said the override would become our first question on any "this slug is slow" report. It becomes the second. The first is "which decode route are you on" — because on the XQA/NVFP4-KV route, PIECEWISE isn't the bug, it's the current upstream instruction. 2. The correction I owe you back: your 4/4 does not clear #49010You offered the non-reproduction honestly and scoped it carefully, so this isn't a challenge to the run — it's a challenge to what a needle ladder can settle. Needle/passkey recall is the test least able to detect this class of defect. On this tracker this week I laid out the KV-quant tail ladder for another reporter: NIAH scores 100% across every cache mode we've measured, bf16 down to a format sitting at 54% tail precision. The tail is where JSON keys, closing braces and tool calls break, and needles average straight over it. The proof case is your own campaign. The V-scale write defect was:
That is the same silent-degradation signature #49010 describes ("no crash, no NaN, fluent short-form output"). So the supportable status is "not reproduced by the test least able to detect it," not "may already be fixed." The FlashInfer 0.6.15 hypothesis is plausible and currently unevidenced, which is why the version cut comes out of the wording until something tail-sensitive backs it. The discriminating test on your rig is the one you already own: the extract / struct / cli packs against an fp8 control on the same envelope. If FULL_AND_PIECEWISE on the XQA route is clean there, that's a real clearance. Meanwhile you have a stronger negative than the needles, and it's already in your 27B table. #49010's mechanism is sharply KV-dtype-dependent — 63–71% acceptance on fp8 collapsing to 5–27% on nvfp4. Your acceptance is flat across exactly that axis: 57.2% fp8, 56.6% nvfp4, 56.3% at 98K, 59.4% at 131K. Four arms inside three points across two dtypes and 2× context. That argues you aren't hitting their failure at all, without needing a FlashInfer version to explain it. Full arithmetic — including the ~1.24× per-step residual that #49891 would fit — is on #613. 3. The methodology point, named because it generalises
That's the whole lesson and it deserves to be stated as a rule rather than an apology: validation has to run on the arm you're recommending, not on a sibling that differs by the flag under discussion. Your needle ladder ran with the override on; the 8-pack that showed quality-neutrality used short prompts, which is precisely the regime #49010 says looks fine. Neither one was wrong — they just weren't about the config in the recommendation. It goes into our bench protocol as an explicit line, alongside the one you gave us last week (an arm that silently tests nothing looks exactly like an arm that tests something and finds no effect). Two contributions to how this repo measures things, from a reporter, inside eight days. 4. Where everything stands, since several of these were promises
To be explicit about the boundary, since it's the discipline the whole thread rests on: your V-scale delta is inert when 5. On the PIECEWISE vs FULL_AND_PIECEWISE runWorth spending the boots. Three asks so it settles more than one thing:
Not asking for it on a timeline. But "PIECEWISE required on the XQA route" versus "PIECEWISE is a 2.7× tax" is currently a fork in the guidance we hand people, and that comparison is the thing that resolves it. |
MTP n=3 under FULL cudagraphs on the SM120 NVFP4-KV route — and the caveat I attached was the wrong mechanismFollow-up to my 07-28 conditioning of the "don't override What was actually brokenvLLM picks FlashInfer XQA for NVFP4 KV on SM120, but never advertised XQA as supporting multi-token speculative decode. MTP n=3 means The real gap: XQA needs a uint16 bit-packed causal mask for Same compiled build, same model, same FP8 KV, same 3-request harness, ~31.2K-token prompts. Only the runtime mask implementation reverted:
The failing lane didn't return a wrong needle — it returned unrelated text about the word "sophisticated". Mask rows for So the docs caveat wants rewording from "FULL is unsafe on the XQA route" to "FULL + multi-token spec decode requires XQA's causal mask; 27B dense: MTP n=3 +
|
| tok/s | draft acceptance | mean accept len | |
|---|---|---|---|
| narrative | 96.89 | 57.0% (5,050 / 8,865) | 2.71 |
| code | 128.71 | 76.3% (2,377 / 3,117) | 3.30 |
Single-stream decode; MTP-off on this config has run ~58–66 tok/s. Code acceptance lands on club-3090's ~74% / ~3.2 reference, narrative is well below it — prose is where this checkpoint's MTP head earns least.
Correctness on the promoted service: 3 concurrent, distinct, cache-busted ~32K prompts, 3/3 cold and 3/3 on identical cached replay, max_running=3, all HTTP 200. I'm not claiming a concurrency scaling result — the tok/s above are single-stream and I don't have a clean sweep yet. What concurrency did do is locate the bug: the pre-fix failures showed up in concurrent waves, not in single-request smoke tests.
Incidentally this settles #613 — MTP-on OOMed at 98K there because FP8 KV was the assumed floor. On a patched writer, 131K runs with MTP n=3 and still leaves 2.39 contexts of headroom.
35B-A3B MoE: MTP holds up under concurrency
The fix carries to the MoE — Qwen3_5MoeMTP resolves and FULL decode graphs capture. Sweep at 5 reps/arm, 2048 tokens/lane, ignore_eos, one boot so weights and graphs are identical across arms: aggregate throughput scales cleanly to max-num-seqs 6 at 822.5 ± 35.0 t/s, and per-stream decode is flat at 140–157 t/s across every arm — MTP costs nothing under concurrency. The cost lands entirely in TTFT: 78 ms at c=1, 401 ms at c=6, then 3.4 s / 7.3 s once requests queue past the cap.
Acceptance is flat under concurrency (0.56–0.62), which is the expected result — batching changes the scheduler, not the draft distribution. The apparent rise with concurrency is prompt content, not batching: a per-prompt control puts code prompts at ≥0.70 and prose at 0.52–0.61, a spread of 0.2506 against an apparent concurrency effect of 0.056, and mix alone predicts every arm within a flat −0.02 offset that is present at c=1 too, where by construction there is no batching.
Arms that asked for 8 and 12 ran at actual 6 with 2 and 6 queued — queueing datapoints, not concurrency datapoints, and labelled as such. Peak system throughput is exactly at max-num-seqs.
One number that should not be read against the above: the 1,008,006 KV tokens at 240.84 tok/s I posted on the 28th (7.69× full 131K contexts, no measurable decode cost against fp8). That arm is max-num-seqs 1, batched 8192, MTP off, thinking off — a different envelope, so 240 vs the ~150 single-stream here is not an MTP regression. It stands as the KV-density result it was.
What it rests on
In the running stack: the NVFP4 V-scale writer fix (vllm#46329, @jethac — load-bearing; without it NVFP4 KV on SM120 is silently wrong, root cause in vllm#50084), #49891 for SM12x enablement and drafter routing, #46324 for PIECEWISE spec-decode capture sizes, #50021 for GDN spec-decode index bounds, plus the local masked-XQA patch. Deliberately not using #49757 — good current-main fix, fails as a v0.25.1 backport, and that says nothing about the PR.
Upstream, and the reason I'm not opening my mask patch as a PR: vllm#49718 (@askliar) is a strict superset of it — XQA on SM90/SM12x, uniform and ragged packed masks, a persistent mask buffer for cudagraph pointer stability, sinks. It's blocked on unreleased FlashInfer 0.6.16 and has no local GPU validation, which is what I can supply, so I've posted the integration note there instead. It does not supersede #49818/#49011 (@0xdespot) — #49718 still gates XQA on FP8 KV, while #49818 carries the NVFP4 dtype gate, the FA2-prefill/XQA-decode split and the 5-D layout fix. Credit stays where the OP put it; I'm a hardware validator there, not the owner.
One of mine, draft: vllm#50336, test-only. test_reshape_and_cache_flash unswizzled both K and V even though the writer stores K linearly, and its assert_close(atol=1.5, rtol=0.5) was loose enough to pass with structurally wrong scale ordering — V at 0.76–0.79 rel-L2 against a ~0.095 baseline, and the test stayed green. That tolerance is why the V-scale bug shipped silently. Adds an explicit layout flag and a 0.15 rel-L2 bound; 24 NVFP4 cases pass on the 5090.
Owed
8-pack /150 on both checkpoints. To be explicit: I don't have one on Qwen3.6-27B-NVFP4 or on the 35B at this configuration. My earlier matched FP8-vs-NVFP4 arms (121.5 vs 122.5) were ThinkingCap-27B and predate MTP + FULL — please don't let those numbers get inherited onto these configs. Running them tomorrow.
Masked-XQA is validated on SM120 only; SM121/Spark untested by me, and the correct gate covers all SM12x minors rather than a blanket capability >= 12.
Uh oh!
There was an error while loading. Please reload this page.
Got real NVFP4 KV cache running on a single 5090 under vLLM — not just NVFP4 weights, the KV cache itself in FP4. Sharing the working config, the numbers, and one bug that will bite anyone else trying this.
Quality test (8-pack /150, thinking-ON) is running today — I'll post scores in a reply.
Hardware / stack
0.25.2.dev1(upstream752a3a5) + SM120 NVFP4-KV routeUpstream vLLM still gates NVFP4 KV to SM100/TRTLLM, so SM120 needs the FA2-prefill / XQA-decode route from vllm#49011. Credit to @gtrak and @0xdespot — that thread is where the gates come from, and a PR is in flight (#44851).
Proof it's actually FP4 KV, not a fallback
Every boot logs:
arch=sm120,decode_backend=xqa, Q/O staying bfloat16 rather than SM100's FP8 staging.FP8 vs NVFP4, same server, same 9.39 GiB KV allocation, only the dtype changed:
e4m3That's the whole point — 1.6× the KV out of the same memory. Close to theoretical 2× minus scale-factor overhead.
Current flags
Env:
MAX_JOBS=2,FLASHINFER_NVCC_THREADS=1,PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False.Result: 262,144 context, KV pool 433,446 tokens (1.65× a full-length request), model 17.86 GiB, ~29.5/32.6 GiB VRAM.
Speeds
Decode (single stream):
Concurrency (this config caps at
max-num-seqs 6, so 8/16 queue):Opened up to
--max-num-seqs 16 --gpu-memory-utilization 0.92it scales near-linearly to 880 tok/s aggregate at 16-way with 0.44 s TTFT. I run 6 for VRAM headroom.Prefill on an agent-shaped load (5 concurrent, 12K shared prefix, 89% prefix-cache hit): 12,700–16,200 tok/s, per-stream decode 57–61 tok/s, inter-token 16–18 ms.
Long context: needle-in-haystack retrieval verified at 246,660 prompt tokens at both 50% and 95% depth.
Two things that cost me hours
1.
--max-num-batched-tokenshas a cliff. 3072–6144 is a flat plateau; 8192 is ~3× worse on agent traffic (prefill 4.7k vs 12–19k tok/s, inter-token 48.8 vs 15–17 ms), reproduced on a clean retest. Interesting because it mirrors what I found tuning-b/-ubon llama.cpp, just shifted. Also a hard floor: at 262K the hybrid GDN page size forcesblock_size 2784, and anything below that refuses to boot withIn Mamba cache align mode, block_size (2784) must be <= max_num_batched_tokens.2. The route crashes under concurrency until patched. Mixed prefill+decode batches hit
AttributeError: 'tuple' object has no attribute 'shape'in the XQA stride check, becausenvfp4_kv_datais a(k_data, v_data)tuple and the FA2 prefill branch rebindskv_cache_permuteto it. Single-stream, reasoning, tool calls and 13K retrieval all pass right before it dies — so smoke tests won't catch it. Fix is to bindprefill_kv_cache/decode_kv_cachelocals instead of rebinding the shared name. Reported upstream on #49011 and #44851.Caveats
<parameter>name>instead of<parameter=name>, so the stock parser returns emptyarguments: {}. 0% well-formed at greedy, so it's baked in, not a sampling issue. Running a lenient parser plugin that rewrites the tag (1/6 → 10/10).Quality scores later today.
All reactions