CPU Big Core Contention

CPU Big-Core Contention (the real decode-stall cause)

A co-resident, idle process can crater ork-driver decode by ~40× — and it is not NPU contention. This page documents the mechanism, the proof, and the fix, because the symptom is easy to misread as "the NPU is single-stream / the submit tax dominates."

Symptom

On RK3588, with the oRKLLM service merely running (dashboard up, no model loaded, NPU load 0%), a llama-bench decode (tg) on the open stack crawls to <0.1 tok/s and often never completes a chunk. Prefill (pp) is unaffected (~83 tok/s either way). Stopping oRKLLM → decode jumps to ~4 tok/s.

It is tempting to blame the NPU ("single-stream", "submission tax"). That is wrong: oRKLLM holds no NPU handle when idle (/dev/dri not open, /sys/kernel/debug/rknpu/load = 0% on all cores).

Mechanism — it's the CPU scheduler, on the big cores

• ork-driver pins its latency-sensitive NPU-submit / worker threads to the big cores (A76, cpus 4–7) via pin_big_core() (disable with ORK_NO_AFFINITY=1).
• A co-resident server's always-on threads (e.g. oRKLLM's Node event loop + its dashboard /ws/metrics polling) also land on a big core.
• Decode is ~200 tiny, latency-bound submits per token. Each one: quantize activation → wake the pinned submit thread (which is asleep, blocked on the NPU) → NPU → dequantize. When another thread shares that big core, the submit thread waits behind it for a scheduler slice on every wake.
• It's driven by wake frequency, not average CPU: a "3.4%-idle" metrics poller that wakes often injects a slice of delay per submit × thousands of submits/decode → ~40×. Prefill is a handful of big throughput-bound submits, so it shrugs the contention off — which is exactly the prefill-fine / decode-dead asymmetry observed.

Proof (RK3588, Qwen3-1.7B-Q8 GGUF, llama-bench tg32, 2026-06-24)

co-resident oRKLLM state	decode tok/s
running, on big cores (default)	<0.1 💀
stopped	3.99
running, pinned to little cores taskset -acp 0-3	4.01 ✅

Three points isolate the cause to oRKLLM's big-core presence, not the NPU. ork-driver's own run_i8 is healthy in isolation (mc_prof M=1: 612 µs 1-core / 219 µs 3-core).

Fix

• Quick (any co-resident process): keep it off the big cores — sudo taskset -acp 0-3 $(pgrep -f <proc>). Decode goes <0.1 → ~4 with the process still running.
• Permanent (oRKLLM): shipped in oRKLLM alpha (perf_governor.js) — inference runs in forked workers, so the orchestration process self-pins to little cores at startup (pinOrchestrationToLittle) and each forked inference worker pins to big cores (pinWorkerToBig); big.LITTLE is detected from cpu_capacity (no-op on uniform-core SoCs). So oRKLLM stays resident for the dashboard and its own inference still gets the big cores.
• Possible ork-driver hardening (not done): raise the submit thread's scheduling priority (nice / SCHED_RR) so it preempts idle pollers even when co-located — more robust than affinity, but needs privileges.

IRQ affinity / core isolation (a distinct tuning lever)

The big-core thread pin above is about where the submit threads run. A separate, often-conflated lever is where the hardware-IRQ handlers run. They want opposite placement:

• Submit threads → big cores (A76): low submit latency (the existing pin_big_core() / ORK_NO_AFFINITY mechanism above).
• HW-IRQ handlers → little cores (A55): steer the rknpu (NPU) and galcore (Mali) hardware IRQs to the A55 little cores so they don't preempt the A76 cores running the NEON int4→int8 prefetch/expand pipeline. An IRQ handler firing on a big core context-switches it mid-compute — exactly what you don't want on the cores doing the inflate/expand work.

So the rule is: submit threads on big, IRQ handlers on little — not a contradiction, two different schedulable entities with opposite needs. Steer IRQs via /proc/irq/<n>/smp_affinity for the rknpu and galcore interrupt lines (mask to cpus 0–3).

This becomes critical once the tile-streamed prefetch pipeline (M2 — see Optimization Roadmap / Heterogeneous Serving) is live, because that pipeline keeps the A76 cores busy expanding/tiling the next slices on the NPU's critical path — an IRQ preemption there directly stalls the matmul feed. (For the current decode-only path the big-core thread contention above dominates; IRQ steering is a saturation-pipeline lever.)

Already-covered (no separate work needed): NPU-IOVA anti-fragmentation does NOT need a separate paged allocator — the IOMMU already maps scattered physical pages → contiguous IOVA, and the M2 tile-streaming uses uniform fixed-size tile buffers (a de-facto slab allocator), so fragmentation doesn't arise.

Takeaways

• Before declaring an NPU/decode regression, check what else is on the big cores. The fix is scheduling, not the NPU.
• Submit threads want big cores; HW-IRQ handlers want little cores — opposite placement, becomes load-bearing under the M2 prefetch pipeline.
• This is distinct from the make test single-command-queue caveat (a wedged submit blocks the next) — that is a real NPU-queue property; this page is about CPU scheduling between live processes.
• See also Performance, the decode-bottleneck finding, and oRKLLM's perf_governor.js.