Repository navigation
Exp 2026 06 13 Thread Count Big Cores
Date: 2026-06-13 · Status: 🏆 BEATS librkllmrt on decode at -t 4 · Path: ggml-ork integration
CPU-affinity pinning fixed the standalone multi-core (mc_prof/rknn_vs_ork prefill 7665→4365 µs/matmul, scaling 1.92×), but llama-bench pp512 stayed ~136. Why doesn't the integration benefit?
Profiled the integration's prefill matmul (ORK_PROFILE=1, real long prompt, affinity ON):
ork profile: prefill (M>1) 193 matmuls, avgM 512.0, run 7596 us/matmul
ork profile: run_multicore 193 calls | setup 796 (10%) submit 6309 (83%) copy 490 (6%)
7596 µs/matmul — the slow number, not the 4365 the standalone gets with the same code. So the ork worker threads are slow inside the integration specifically. Cause: llama.cpp's CPU threadpool busy-waits (spins) on all 8 cores while the single NPU matmul node runs, starving the 4 ork NPU-driver threads (calling thread + 3 pinned workers) that need the big cores. Plus the 4 little A55 cores in an -t 8 pool drag every CPU-side op (the threadpool waits on its slowest thread).
Qwen3-1.7B-w8a8, llama-bench, warm, DDR 2112 / CPU 2.4 GHz. Sweeping -t (CPU thread count):
-t |
pp512 (prefill) | tg128 (decode) |
|---|---|---|
| 8 (default, 4×A76+4×A55) | 136.2 | 9.24 |
| 6 | ~156 | 10.28 |
| 4 (big A76 cluster only) | 155.1 | 12.84 |
| 2 | 132.8 | — |
| librkllmrt (closed, same model) | 183.9 | 11.71 |
-
-t 4(big cores only) is optimal for both phases: prefill 136→155 (+14%), decode 9.24→12.84 (+39%). -
Decode at
-t 4(12.84) BEATS librkllmrt (11.71) — the open stack is faster than the closed runtime on token generation when it isn't dragged by the little cores. Prefill reaches 84% of librkllmrt (155 vs 184). - The integration's prefill was never NPU-matmul-bound — it was CPU-thread contention + little-core drag. The affinity fix is necessary (workers must be on big cores) but insufficient while llama.cpp also spins 8 threads across all cores.
-
-t 8is a poor default on big.LITTLE: the 4 slow A55 cores hold back the threadpool barrier and fight the NPU-driver threads for the big cores.
Tried making it automatic: pin the calling/main thread to the big cluster at ork_npu_init (before llama.cpp spawns its threadpool, so the pool inherits the mask — the taskset -c <big> effect without the flag). Measured (no taskset):
-t |
pp512 | tg128 |
|---|---|---|
| 4 | 156.6 | 12.72 (✓ unchanged — already optimal) |
| 8 | 132.3 | 2.28 (✗ cratered) |
It doesn't help -t 4 (already optimal) and craters -t 8 — restricting 8 threads onto 4 cores oversubscribes 2:1 and thrashes (taskset's freely-migratable set behaves differently from our inherited-mask + single-core-worker mix). Reverted (ork-driver b9a5c4e); the driver pins only its own dedicated NPU-driver threads, never the whole process. Thread count is the user's lever.
🏆 Run the open stack with -t <big-core-count> (4 on RK3588), not -t 8. That alone makes decode beat librkllmrt and lifts prefill to 84% — and forcing it from the driver backfires (above), so it stays a serving/CLI choice. The remaining prefill gap is ork's single-core large-M matmul kernel (3.3× slower than RKNN's per core — see Matmul apples-to-apples), now the main lever. The proper automatic fix is heterogeneous scheduling — the CPU threadpool should sleep (not spin) while the NPU matmul runs, and the thread count should match the big cluster — but that belongs in the llama.cpp threadpool / serving layer, not a process-wide affinity hammer. See Optimization Roadmap.