Negative results: 19 MoE-offload optimizations reviewed against current dev — what's already in the tree, what doesn't transfer, what's sub-noise

[KingIcyCreamProjects]

Short version of a larger review I ran against current dev — each idea was first checked against the code (is it already in the tree?) and then attacked on impact for a single-stream, disk-bound GLM-5.2 config (9950X3D + RTX 5090, int4 experts on NVMe, ~800 ms/token budget at 1.25 tok/s). Posting the negative results so these don't get re-proposed one at a time.

Already in the tree (no need to build):

• speculation-guided expert prefetch: the MTP verify pass runs pilot_prefetch per layer over all draft positions (c/glm.c:3396-3397), and PILOT_REAL=1 turns those into real async loads (see [Feature]: Pre-fetching MoE experts #333 for the nuance about the draft forward itself)
• draft-position expert union/dedup in moe() FASE B (:2965-2970)
• resident-expert compute overlapping miss I/O: misses dispatch async up front and the matmul loop waits lazily per-expert — the full block drain only exists in the Metal arm (#ifdef COLI_METAL)
• grouped CUDA expert dispatch with gate+up fusion default-on (grouped_hidden_w4_dual, COLI_CUDA_DUAL_PROJ=1)
• cache-aware routing (CACHE_ROUTE) and learned co-activation prefetch (COUPLE)

Mechanism doesn't transfer to this router:

• softmax-mass "confidence filter" prefetch (arXiv 2502.12224's 79%→97%): the router is sigmoid+bias — per-expert scores don't sum to 1, so the paper's mass threshold turns into a fixed ~64-expert flood per position (~56 false positives at topk=8) that thrashes the LRU. Same failure family that made PILOT_K default drop to 6 under real loads.

Killed by measurements already in the repo:

• per-miss H2D streaming of cold experts to GPU: the 6×5090 experiment doc already measured PCIe staging losing end-to-end (5.39–5.44 vs 6.0–6.28 tok/s baseline) — on hardware where it should look best
• any verify-pass expert cap: the EXPERT_BUDGET quarantine numbers (MTP acceptance → 0%, 14.7 experts/layer loaded vs topk=8 — more I/O, not less)
• frequency-weighted ecache eviction: the predicted-probability signal doesn't exist for cached experts at the eviction site (collapses to LFU), and it would preferentially evict freshly PILOT-staged experts (freq≈0) — attacking the prefetcher's wins; frequency already runs the pin tier (tier.h LFRU)

Real but sub-noise at batch-1 decode (Amdahl): pilot-worker condvar (the 200 µs poll only fires on an empty ring — worst case 0.025% of a token), router top-K heap (~0.1 ms/token, plus a tie-breaking hazard that would silently shift routing), LRU victim heap/CLOCK (the scan is a few L1-hot cache lines guarding an NVMe read 1000× its cost), residency bitmap for expert_is_resident (per-layer npin is ~6, not the ~469 model-wide total), .qs scale-read coalescing (the three scale tensors are file-contiguous — I checked the shards — but it's 2 syscalls per miss, ~0.1% of the miss's own cost), DSA scratch hoisting (the select block is gated nk > index_topk from the model config — 2048 on my GLM-5.2 snapshot — so it never even runs at 96/256-token replays).

What survived review as worth someone's time: the dist_build full-vocab qsort per sampled token (measured 5.6–8.0 ms → ~0.78 ms with a partial top-p select; filed separately with an implementation branch) and a pin-vs-ecache hit-telemetry split as the gate for any eviction-policy work including #223 (also filed with a branch).

Full writeup with line refs and per-idea verification available if anyone wants it.