fix: size the curated coding-agent context from the model's actual KV geometry - #5915
Merged
Conversation
… geometry
The "Recommended coding-agent setup" card offered a 48 GB Apple Silicon
machine a 64K launch context with only a vague headroom sentence to justify
it. The number was too small and nothing in the code let a reader check it.
Qwen3.8-27B holds a KV cache on only its 16 full-attention layers (the other
48 are Gated DeltaNet, whose state is constant in the window), which costs
65.5 KB per token at a bf16 cache — the same per-token figure the SGLang
recipe sizes its pools from. Against the ~75% of unified memory macOS gives
the GPU, and the 15 GB MLX 4-bit weights:
48 GB -> 64K reserved 19 GiB of 36. 128K reserves 23 GiB, still leaving
~13 GiB for PortOS, the harness and local image/video work.
64 GB -> 256K, the checkpoint's own ceiling, reserves 31 GiB of 48.
128 GB -> the same 256K on the quality checkpoint, with budget to spare.
So each Apple tier moves up one step. A 1M-token window is not reachable on
any of them: its KV cache alone is 65.5 GiB, more than a 48 GB machine has in
total, and the catalog caps this checkpoint at 256K regardless (the
1M-context local model there is Nemotron 3 Nano 30B-A3B).
Profiles now carry contextTokens rather than a display string, the card
renders the label from it, and each note states the arithmetic and the
kv-quant lever that buys more. A parameterized test pins weights + KV under
each tier's GPU budget with an 8 GiB reserve, so the next edit cannot promise
a window the machine will not load.
Claude-Session: https://claude.ai/code/session_01FVBPGvC77m8KA6J4UwDRnv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Recommended coding-agent setup card offered a 48 GB Apple Silicon machine a 64K launch context, justified only by a vague "practical unified-memory headroom" sentence. Double-checking that number against the model's real geometry showed it was too conservative — and that nothing in the code let a reader verify it either way.
Qwen3.8-27B holds a KV cache on only its 16 full-attention layers (the other 48 are Gated DeltaNet, whose state is constant in the window). That costs 65.5 KB/token at a bf16 cache — the same
KV_KB_PER_TOKENserver/lib/sglangQwenRecipe.jssizes its pools from. MTPLX reserves the window up front (--context-window), so it is a memory reservation, not a free ceiling. macOS gives the GPU ~75% of unified memory; the MLX 4-bit weights are 15 GB.1M is not reachable on any of them. Its KV cache alone is 65.5 GiB — more than a 48 GB machine has in total, before the weights. And the catalog caps this checkpoint at 262,144 tokens regardless; the 1M-context local model there is Nemotron 3 Nano 30B-A3B, not this one. 256K at 48 GB would leave 4.6 GiB, which is why that tier stops at 128K.
Changes
HardwareLlmRecommendation.jsx— profiles carrycontextTokensinstead of a display string; the card renders the label from it. A header comment states the geometry, the budget, and where both numbers come from. Each tier's note now gives the arithmetic, and the--kv-quant q8/--cache-type-k q8_0lever that buys a longer window.LocalLlmTab.test.jsx— a parameterized test pins weights + KV under each tier's GPU budget with an 8 GiB reserve, so a future edit cannot promise a window the machine will not load.The RTX 3090 profile keeps its 64K (weights + KV already claim ~19 of 24 GB) and now says so.
Test plan
cd client && npx vitest run— 807 files, 10,002 tests, all green.https://claude.ai/code/session_01FVBPGvC77m8KA6J4UwDRnv