Proposal: decode speed estimate for MoE expert offload residencies

[beatakouchnir]

Problem

#3676 shows which residency fits the memory ceiling, but not how fast the model will decode there. Users find out only after downloading and loading checkpoints that are often hundreds of GB.

Proposal

Show an estimate next to the fit hint for families with a measured routing curve, e.g. Fit to memory: 41.4% resident · ~99 GB · ~12 tok/s (estimate); no estimate otherwise. The model comes from boyle, the runtime I measured offload with before upstreaming it in #2595:

• Hit-rate curve per family: routed expert ids recorded during decode, replayed through a per-layer LRU cache at 14 residency fractions (oMLX's ExpertCache is also per-layer LRU). Curves were identical at 4-bit and 8-bit.
• Step time = compute floor (one measured anchor per family) + misses per token × expert bytes / drive bandwidth (a ~3 s parallel random-read probe, cached). Reported as a band.

Evidence (boyle's runtime, out-of-sample)

model	machine	budget	estimate (tok/s)	measured
Qwen3-235B-A22B-4bit	M5 Max 128 GB	0.45 of experts	12.5 (±25%)	11.7
Qwen3-30B-A3B-4bit	M1 Pro 32 GB	12 GB	14.7 (11.8–18.4)	14.5
Qwen3-30B-A3B-4bit	M1 Pro 32 GB	20 GB	19.3	17.5 (in band)

Known miss: Qwen3.5-397B at its anchor point measured 7.2 tok/s through the HTTP server, below the 7.6–11.9 band; the model does not include serving overhead. Trace replay predicted live hit rates within 1–3 points on four configurations. Curves exist for five families (qwen3_moe, qwen3_5_moe, glm_moe_dsa, gemma4, qwen4_exp), under 4 KB of JSON in total. Compute anchors would be re-measured on oMLX.

Steps (one PR each)

1. Decode-only miss counters, following feat(moe): report expert offload hit/miss counters in model status #3910, so live hit rates can be checked against a curve.
2. An opt-in routing recorder plus a script that turns a recording into a curve.
3. The estimate in model settings and /api/models.

Questions

1. Is a small per-family curve file OK in the repo? If not, steps 1 and 2 still stand alone.
2. Drive bandwidth: a first-use probe, or derived from observed fetch times after a model has run?
3. Faster expert SSD offload: single-stream scheduling, fetch/compute overlap, lookahead prefetch (ported from streamlx) #3686's fetch/compute overlap makes a step cost closer to max(compute, fetch) than their sum. Should I anchor after it lands?