Skip to content

Exp 2026 06 27 Prefetch Overlap and Stream Pool

Jump to bottom
Michael Fischer edited this page Jun 28, 2026 · 2 revisions

Prefetch-Overlap Measurement + RAM-Resident Stream Pool

Commit: ork-driver a443b64 (v0.5.0) · probe tools/stream_prefetch_probe.c · API ork_stage_* / ork_stream_pool_* / ork_mm_load_i4a8_import in src/npu.c + include/ork_npu.h. RK3588 10.3.0.236, perf governors, taskset 4-7, median 15 reps, all bit-exact vs CPU int8 reference.

TL;DR

To stream int4-stored weights, each swap is FILL (int4→int8 inflate + tile into a non-mapped staging dma-buf + cache-clean) → MAP (bare MEM_CREATE import → IOVA) → SUBMIT, and a prefetch thread can run FILL(N+1) behind SUBMIT(N). Measured the pieces. MAP is cheap (48–169 µs); MEM_DESTROY UNMAP is expensive (~0.5–2 ms) — the irreducible swap killer. So a transient unmap-every-swap ring loses (overlap never reaches t_submit; fill only hides at M ≳ 256). But the measurement validates the RAM-resident-cache design: since unmap is the dear op, cache the inflated int8 in RAM, keep hot entries mapped, and unmap only on eviction — a hit then pays just the cheap map and never re-inflates. Built that as ork_stream_pool_*.

The measurement (stream_prefetch_probe.c)

op cost note
map (MEM_CREATE import of a filled staging buf) 48–169 µs cheap; isolated from fill
unmap (MEM_DESTROY) 461–1827 µs (~0.5–2 ms) expensive — irreducible per-swap if you unmap every swap
fill (int4→int8 inflate + tile) 415 µs (1 MB expert) / 2.7 ms (4 MB backbone) M-independent
int8 fill-once bare-map-only 170 µs @4 MB / 654 µs @16 MB confirms map ≪ fill
  • Overlap vs serial (prefetch FILL behind SUBMIT): only ~1.0–1.17× at multi-core; the overlapped wall/swap never reaches t_submit because unmap sits on top of every swap.
  • Fill-hide crossover ≈ M 256 (single-core backbone): the inflate fully hides once t_submit ≥ ~2.5× t_fill (M=256/512 → "FILL HIDDEN"); at decode M it does not.

Why this confirms the cache design

The expensive op is unmap, not map and not (on a hit) inflate. So the correct structure is a memory hierarchy, not a transient ring:

int4 on disk/RAM (compact)
  → inflated int8 in RAM   (LRU, big budget — pays inflate ONCE, avoids re-inflate)
    → IOVA-mapped hot tier (≤4 GiB; evict RARELY because unmap is ~0.5–2 ms)

A hit pays only the cheap map (or nothing, if the entry is still mapped) and zero re-inflate. Unmap is amortized over many hits at eviction time. This is the routing-skew hit-rate lever realized in RAM budget (8–15 GiB) instead of the 1.2 GiB usable IOVA that capped the earlier MoE cache — without paying re-inflate on hits.

What was built (v0.5.0, all bit-exact, make test green)

  • ork_stream_pool_* — RAM-resident inflated-int8 cache. add_i8 (copy pre-tiled bytes) / add_i4a8 (inflate int4→int8 once); cheap map/run/unmap; entries persist in RAM across unmap; remove = caller-driven evict; entry_bytes for the RAM budget. Lifecycle only — the LRU/eviction policy stays with the caller (ggml-ork).
  • ork_mm_load_i4a8_import — int4a8 import sibling of ork_mm_load_i4a8 (the i8 one shipped in a443b64's predecessor).
  • ork_stage_* — the staging-ring primitives the probe exercises (fill/map/run/unmap/free).

Verdict

✅ The RAM-resident inflated-int8 cache with cheap map-on-hit and evict-time unmap is the right streaming structure — and the measurement is what ruled out the simpler unmap-every-swap ring (MEM_DESTROY ~0.5–2 ms). ⚠️ Wins are M>1 prefill/verify (fill hides ≥ M256) and warm cache hits (cheap map, no re-inflate); it does not help M=1 decode (fill un-hidden + bandwidth-bound) or any unmap-per-swap pattern, and it does not escape the 4 GiB cap. NEXT: Step 2 — retarget ggml-ork's hot-expert/wcache LRU to budget RAM-int8 and map-on-hit via the pool, keeping hot entries mapped (don't unmap per swap), then measure end-to-end on a streamed M>1 path.

Update — end-to-end in ggml-ork (Step 2 + 2b)

Wired the pool into ggml-ork's streaming LRU (board commit d2e8526e8, behind ORK_STREAM_POOL=1, off by default; submodule → ork-driver a443b64). Bit-exact perplexity, zero regression at the default (resident) budget. But the end-to-end decode verdict is a clean negative: forced-stream 1.7B decode (-n 8, IOVA 1536 MB, RAM 22 GB) is identical pool-on vs pool-off (0.03 == 0.03 t/s; resident ceiling 3.85). ORK_PROFILE root cause: run_i8 (NPU compute) is ~0.6% of the decode wall; ~99% is per-token IOVA map/unmap churn (~100–170 weights cycled through the window every token, each MEM_CREATE+MEM_DESTROY). The inflate+copy the pool removes is dwarfed by that churn → it changes nothing at M=1. This is the 4 GiB cap as churn: a decode working set larger than the window must unmap-to-map every weight every token. (A wiring gap also left remaps=0 — the grouped QKV/gate-up handler bypasses the pool's cheap-remap branch — but even fixed, unmap churn dominates.)

Conclusion (banked): NPU-streamed decode is churn-bound — big-model decode runs on CPU (native Q4_K, fused, no churn); the NPU's lane is resident-hot weights + M>1 prefill/verify. The pool's one open upside is M>1 prefill streaming (import removes the copy, prefetch hides the inflate at M ≳ 256 — both proven at the microbench level), still to be demonstrated end-to-end on a real >4 GiB dense transformer (the board has none yet — only Q4 MoE 27B + hybrid-SSM 9B). Env gotchas: pool needs ulimit -n raised (each entry holds Sk×Sn+Sn dma-buf fds); check sudo-proc liveness with ps -p, not kill -0.

Clone this wiki locally