roadmap: SUPERSEDED/REFUTED banners on Rank 1, Rank 2, and phase-aware scheduling All three predate the 2026-08-19 measurements. Rank 1: native W4A4 compute is ~54x the W8A8-inflate route; what shipped is int4/NF4 STORAGE + int8 compute (and ORK_QUANT=4 forcing dense int4 hits the multi-domain IOVA wall). Rank 2: overlap is dead for MoE (CPU ~6% of NPU GEMM time), OMP_WAIT_POLICY=ACTIVE flat, the sleep/futex variant harmful. Phase-aware: prefill won by moving experts TO the CPU (NPU matmul is 11.6% of the wall) and decode bandwidth does not sum (1big+2little slower than 3big; every NPU-participating decode config lost to all-CPU) — phase-awareness is right, the engine assignments are inverted.
roadmap: 4c — shipped knob-free MoE auto-profile (1.59x native prefill) + open backend items Records the shipped result with its trade (+5.5% PPL, ~15% slower decode), the decode ladder, the ORK_ATTN correctness fixes (kept off — perf-neutral), and the open items owned by this repo: the res=-3 pp512 blocker, the ORK_ALLOW_Q4_SRC probe for the Q4_K ssm_* projections (31% of CPU-backend prefill), the uncommitted ORK_OPPROF profiler, and folding the two scratch WIP docs. Cross-links the driver items (ork-driver Tier 14) and the model-recipe items (oRKLLM orkpack conversion).
roadmap: M2 end-to-end decode loses on RK3588 (~2.9x); default-off setting + RK3588 disclaimer; lever is M>1
roadmap: MoE decode partition + corrected submit-floor verdict (resident expert beats CPU at M=1)
wiki: 2026-06-26 — MoE-on-NPU M1a (expert persist/load + IOVA wall) & M3 (routing-skew hot-cache sizing) milestones
wiki: 2026-06-26 — MoE-on-NPU build milestones (M0 viable) + newly-accepted opt items M0 sparse-MoE expert-GEMM proven VIABLE on NPU (beats CPU 3-10x, no crossover at per-expert M>=8; direct int4->int8 inflate closed the f32-repack wall 16-30x, bit-exact). Staged M0->M1(MUL_MAT_ID expert-persist)->M2(tile-streamed double-buffer)->M3(hot-expert decode cache). Added dynamic draft sizing, radix-tree prefix caching, paged KV (PagedAttention), IRQ affinity (rknpu+Mali to A55), and a contingency/deprioritized list (Lookahead/Jacobi, Mali RoPE/SwiGLU, NPU-IOVA paged allocator).
wiki: 2026-06-26 saturation findings — mixed orkpack, DMA-fill chain, phase-aware scheduling, fit-by-partition residency Roadmap: add saturation roadmap threads (mixed int8/int4-NF4 .orkpack DONE+PPL-validated; DMA-buf fill optimization chain DONE; phase-aware heterogeneous scheduling DESIGN). Residency: add UMA reality (NPU buffer is system RAM, runs int8; int4 win is disk/GPU-CPU not NPU resident) + fit-by-partition (~35B mixed fits resident across NPU<=4GB int8 + CPU/GPU ~28GB int4, no streaming). ggml-ork changes on board branch feature/mixed-orkpack-v2 (9f2d659, 2e086b4, 15189076d, 8bed494).
2026-06-24 correction banners: int4 W4A4 not-yet-viable (downgrade Rank-1 framing); decode-stall is CPU big-core contention not NPU single-stream; GGUF-Q8 vs w8a8/.rkllm path rule. Cross-link ork-driver wiki.
docs(wiki): document multi-turn decode performance resolution
Document CPU NEON dequantization optimization and June 20th prefill stability findings
Incorporate Qwen3-0.6B benchmarks and memory bandwidth saturation dynamics into optimization roadmap
Update roadmap GPU strategy to focus on KV Cache compression
Incorporate KV Cache compression and heterogeneous GPU splitting insights into roadmap
Update SBC Performance Optimization Roadmap with RKLLM comparisons and RAM bandwidth profiles
Add SBC performance optimization roadmap page
docs: document physical path integration gaps and performance roadmap
doc: add project summary, residency bug, and ccache optimizations to wiki
Initial Home page