glm: overlap the indexed full-layer prefill sweep with a pread prepare of the next layer#528
Open
andreaborio wants to merge 3 commits into
Open
Conversation
The indexed generation prefill hardcoded full_layer_prefill=false in its per-layer mapping and forced the runtime flag off, so every chunk took the selected-expert addr path regardless of size. A chunk above ~64 tokens selects more unique experts per layer than the cache holds; the loader's overflow branch then addresses experts straight out of the mapped model, but decode-only spans do not cover the routed expert tensors, and prefill died with "failed to map overflow expert views". Net effect: every real generation prompt above ~64 tokens failed, which went unnoticed because benchmarks used short prompts and the perplexity flow stayed under the overflow threshold. Compute the same full-layer threshold the plain prefill flow uses (glm_graph_stream_prefill_full_layer_enabled) and pass it to both the layer mapping and the runtime flag: big chunks now stream full layers with the mm_id kernels (measured prefill 1.5 -> 9.4 t/s on a 795-token prompt, and it completes instead of failing), short prompts keep the addr path unchanged. Decode is untouched (decode-consistency stays bit-exact); scoring a text under the overflow threshold shifts +0.7% ppl from the documented addr-vs-mm_id accumulation-order difference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When the expert cache was at budget, prepare_selected_batch spilled every
uncached expert to whole-tensor overflow views instead of evicting. The
overflow wrap needs the routed expert tensors in the mapped view set, which
the decode-expert-cache prefill mapping does not provide, so any small-chunk
prefill against a full cache failed ("failed to map overflow expert views")
— most visibly checkpoint-extend prefills, which broke the disk KV cache
flow right after it loaded a matching checkpoint.
Reuse-evict at budget exactly like the decode loader does: commands are
synced by the caller before prepare, so reuse cannot race the GPU, and
prepare_load_buffers protects the layer's unique selected set. Overflow
views remain as the last resort for allocation failure.
With this plus the indexed-prefill full-layer threshold fix, the disk KV
cache works end to end on GLM 5.2: a 321-token request costs 181 s cold and
32 s on a same-prefix follow-up (checkpoint loaded in ~20 ms, 14-token
suffix prefill), including across a server restart (33 s, disk checkpoint).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e of the next layer The indexed prefill maps each full layer and then pays the whole expert sweep as GPU demand faults inside the kernels: measured 2.6 GB/s on an M5 Pro 64GB against the ~7 GB/s the SSD sustains with batched reads. Reuse the existing full-attention layer prepare machinery (pread pool, one layer of lookahead) around the indexed map so the next layer is warmed while the current one computes. Measured on GLM 5.2, paired runs: 271-token prompt prefill 3.5 -> 8.5 t/s, ds4-bench sweep 2048..8192 prefill 1.6-2.0x at every frontier, generation unchanged, greedy output byte identical. Opt-in via DS4_METAL_ENABLE_GLM_INDEXED_PREFILL_PREPARE=1 while it soaks.
andreaborio
pushed a commit
to andreaborio/ds4
that referenced
this pull request
Jul 10, 2026
…gn claims Re-ran every claim from the campaign handoff on the same M5 Pro 64GB box: C1 (glm5.2 line regresses DS decode ~2.8x), C2 (GLM indexed prefill prepare x2.5, PR antirez#528), C4 (router readback = ~40% of streamed GLM decode token), C5 (the four opt-in patches are default-neutral, code audit) and C6 (GLM MTP nextn acceptance 55%, NO-GO) all confirmed. C3 (advisory per-miss OFF as a DS chat config) does NOT reproduce at steady state and is downgraded to a cold-cache effect. Includes the corrections found (stale base SHA in the handoff, unpushed RAM-guard commit 7c9f033) and the pre-publication edits needed on the two upstream drafts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
andreaborio
pushed a commit
to andreaborio/ds4
that referenced
this pull request
Jul 10, 2026
Add a what-differs-from-upstream callout at the top of the README (five mainline additions, the GLM 5.2 SSD-streaming line with PRs antirez#520/antirez#528, verification docs, open upstream PRs) and replace the stale experimental-GLM paragraph with the current state of the line, including the two measured caveats (DS decode regression on upstream glm5.2, MTP NO-GO). FORK_NOTES: add the RAM-guard row (branch fix/refuse-oversized-resident-maps, PR-ready) and an honest per-component status row for the GLM line. All fork-facing claims fact-checked against git history, the measured CSVs and the PR bodies by an adversarial review pass; scoped the re-verification claim (sweep figures are antirez#528's), fixed a dead anchor, disclosed the full branch content. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On SSD streaming, the GLM indexed prefill maps each full layer and then pays the whole expert sweep as GPU demand faults inside the kernels: an M5 Pro 64 GB reads about 2.6 GB/s that way, against the ~7 GB/s the same SSD sustains with batched reads. The full-attention prefill flow already hides this behind its layer-prepare machinery (a pread pool that warms the next layer while the current one computes); the indexed flow never got it.
This reuses that machinery around the indexed per-layer map: join the prepared layer at the top of the iteration, start the prepare of layer il+1 right after mapping il, join-all at chunk end. One layer of lookahead, 48 lines in ds4.c, no new mechanism. Opt-in for now behind
DS4_METAL_ENABLE_GLM_INDEXED_PREFILL_PREPARE=1, default behavior unchanged; fine flipping it to default once it has soaked on more configs.Sits on top of #520: before it the indexed flow hardcodes
full_layer_prefill=falseand never reaches this path, so the branch carries those two commits. If #520 lands first this rebases to the single commit.Testing
Machine: Apple M5 Pro 64 GB, macOS 26 (Darwin 25.5.0), Metal,
make clean && make,git diff --checkclean. Same GLM 5.2 artifact and ds4-native layout note as #520; the change only drives the existing prepare machinery and never inspects tensor dtypes.Correctness:
./ds4_test --server,./ds4_test --metal-kernels— OKDS4_TEST_SSD_STREAMING=1 DS4_TEST_MODEL=DeepSeek-V4-Flash-IQ2XXS-...--logprob-vectorsand--metal-ssd-streaming-cache-pressure— OK (the change is inside the GLM indexed prefill loop and env-gated, DeepSeek logits untouched)Speed. Paired short-prompt runs, 271 tokens, where the sweep dominates (the common agentic case): prefill 3.5 -> 8.5 t/s, decode unchanged (0.9 first-tokens / ~2 steady either way). ds4-bench sweep on promessi_sposi.txt, back to back:
Prefill 1.6-2.0x at every frontier; the small generation gains are within this box's run-to-run variance.
Two notes.
make cpufails on current glm5.2 (bd89932) with or without this patch (glm_graph_normal_layer_countis called from CPU-compiled code but defined under the GPU guard); pre-existing, can send a one-liner separately if useful. No overlap with #514: that changes how expert bytes are laid out on disk, this overlaps the existing reads with compute, and they should compose.