From ee026414920f8a2527c5a1f1eb1e362f43405fb6 Mon Sep 17 00:00:00 2001 From: Andrea Borio Date: Wed, 8 Jul 2026 11:27:35 +0200 Subject: [PATCH 1/2] glm: honor the full-layer prefill threshold in the indexed prefill flow The indexed generation prefill hardcoded full_layer_prefill=false in its per-layer mapping and forced the runtime flag off, so every chunk took the selected-expert addr path regardless of size. A chunk above ~64 tokens selects more unique experts per layer than the cache holds; the loader's overflow branch then addresses experts straight out of the mapped model, but decode-only spans do not cover the routed expert tensors, and prefill died with "failed to map overflow expert views". Net effect: every real generation prompt above ~64 tokens failed, which went unnoticed because benchmarks used short prompts and the perplexity flow stayed under the overflow threshold. Compute the same full-layer threshold the plain prefill flow uses (glm_graph_stream_prefill_full_layer_enabled) and pass it to both the layer mapping and the runtime flag: big chunks now stream full layers with the mm_id kernels (measured prefill 1.5 -> 9.4 t/s on a 795-token prompt, and it completes instead of failing), short prompts keep the addr path unchanged. Decode is untouched (decode-consistency stays bit-exact); scoring a text under the overflow threshold shifts +0.7% ppl from the documented addr-vs-mm_id accumulation-order difference. Co-Authored-By: Claude Fable 5 --- ds4.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/ds4.c b/ds4.c index edf566489..836084821 100644 --- a/ds4.c +++ b/ds4.c @@ -30722,7 +30722,15 @@ static bool glm_graph_forward_indexed_tokens( const uint32_t drain_interval = progress_flush_interval != 0 ? glm_graph_indexed_prefill_drain_interval() : 0u; const bool progress_requested = display_progress && work_total > 0; - ds4_gpu_set_glm_streaming_prefill_full_layer(false); + /* Mirror the plain prefill flow: above the full-layer threshold the + * selected-expert addr loader is the wrong tool — a chunk of N tokens + * selects far more unique experts per layer than the cache can hold, and + * its overflow branch needs the full expert tensors mapped. Hardcoding + * false here broke every generation prompt above ~64 tokens ("failed to + * map overflow expert views"), while short prompts keep the addr path. */ + const bool full_layer_prefill = + glm_graph_stream_prefill_full_layer_enabled(g, n_tokens); + ds4_gpu_set_glm_streaming_prefill_full_layer(full_layer_prefill); if (trace) { glm_graph_indexed_prefill_tracef( @@ -30818,7 +30826,7 @@ static bool glm_graph_forward_indexed_tokens( weights, il, n_tokens, - false); + full_layer_prefill); if (ok) ok = ds4_gpu_begin_commands() != 0; } const ds4_layer_weights *l = &weights->layer[il]; From b85f6f8ea5429119d8a23c2317ff40151ba22619 Mon Sep 17 00:00:00 2001 From: Andrea Borio Date: Wed, 8 Jul 2026 11:44:08 +0200 Subject: [PATCH 2/2] glm: evict at budget in the batch selected-expert prefill loader MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When the expert cache was at budget, prepare_selected_batch spilled every uncached expert to whole-tensor overflow views instead of evicting. The overflow wrap needs the routed expert tensors in the mapped view set, which the decode-expert-cache prefill mapping does not provide, so any small-chunk prefill against a full cache failed ("failed to map overflow expert views") — most visibly checkpoint-extend prefills, which broke the disk KV cache flow right after it loaded a matching checkpoint. Reuse-evict at budget exactly like the decode loader does: commands are synced by the caller before prepare, so reuse cannot race the GPU, and prepare_load_buffers protects the layer's unique selected set. Overflow views remain as the last resort for allocation failure. With this plus the indexed-prefill full-layer threshold fix, the disk KV cache works end to end on GLM 5.2: a 321-token request costs 181 s cold and 32 s on a same-prefix follow-up (checkpoint loaded in ~20 ms, 14-token suffix prefill), including across a server restart (33 s, disk checkpoint). Co-Authored-By: Claude Fable 5 --- ds4_metal.m | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/ds4_metal.m b/ds4_metal.m index e8403cf6a..a666fcdaa 100644 --- a/ds4_metal.m +++ b/ds4_metal.m @@ -12859,12 +12859,16 @@ static int ds4_gpu_stream_expert_cache_prepare_selected_batch( continue; } - if (cache_budget != 0 && reserved_entries >= cache_budget) { - unique_entries[u] = NULL; - continue; - } - - const int force_reuse = 0; + /* At budget, evict-and-reuse like the decode loader does instead + * of spilling to whole-tensor overflow views: the overflow wrap + * needs the expert tensors in the mapped view set, which the + * decode-expert-cache prefill mapping does not provide, so a + * full cache made any checkpoint-extend or small-suffix prefill + * fail ("failed to map overflow expert views"). Commands were + * just synced by the caller, so reuse cannot race the GPU, and + * prepare_load_buffers protects this layer's unique set. */ + const int force_reuse = + cache_budget != 0 && reserved_entries >= cache_budget; ds4_gpu_stream_expert_readahead_range(unique_gate_offsets[u], gate_expert_bytes); ds4_gpu_stream_expert_readahead_range(unique_up_offsets[u],