Repository navigation
Multi Turn Decode Performance Slowdown Resolution
Note (2026-06-24): A separate decode-stall cause has since been isolated — CPU big-core scheduling contention from a co-resident process (e.g. an idle oRKLLM whose orchestration threads sit on the big cores and preempt ork-driver's pinned NPU-submit thread), distinct from the weight-residency / op-routing issues documented here. If decode is slow while a co-resident server is up, check CPU affinity first — see CPU Big-Core Contention.
During multi-turn conversational chat sessions on Rockchip SBCs (like the Radxa ROCK 5B powered by the RK3588), a severe performance anomaly was discovered. While the first turn (initial prompt prefill and first token generation) executed at full NPU/CPU speed, the second turn (and subsequent chat continuation turns) suffered a catastrophic slowdown.
On plain Transformer models (such as Qwen3-1.7B-UD-Q4_K_XL) and hybrid/recurrent architectures (such as LFM2.5-8B-A1B-UD-Q4_K_XL), decode throughput dropped dramatically, and with prefix caching enabled, users would occasionally experience completely blank output or massive decode latency spikes.
We isolated, resolved, and verified the two independent bottlenecks responsible for this performance drop. After applying targeted optimizations in llama_memory_recurrent and the context reservation scheduler, multi-turn decode performance maintains 100% parity across all turns, restoring a fluid, real-time chat experience.
Tip
Key Optimization Achievements:
- Qwen3-1.7B-UD-Q4_K_XL: Turn 2 decode speed maintains 17.8 t/s (matching Turn 1's 17.9 t/s exactly; ratio of 1.01×).
- LFM2.5-8B-A1B-UD-Q4_K_XL: Turn 2 decode speed maintains 18.5 t/s (matching Turn 1's 19.0 t/s; ratio of 1.03×).
- Latency Overhead: Eliminated multi-second stalls and graph re-reservation pauses, achieving flat-line parity (~1.0×) between turns.
The investigation revealed that the slowdown was caused by two entirely separate architectural bottlenecks in the main llama.cpp inference loop, which compounded during multi-turn continuations.
For hybrid and recurrent architectures (like Mamba, RWKV, and LFM2.5), the cell cache tracks which physical cache cells store the recurrent hidden states for active sequences. During batch assembly and re-ordering in llama_memory_recurrent::find_slot, the runtime swaps sequence tails to align logical and physical sequence indices.
The previous implementation performed a full-size loop scanning every cell in the recurrent memory cache (which scales directly with the full context size
// OLD LOOP: Scales O(size_of_cache) per token
for (uint32_t j = 0; j < size; ++j) {
int32_t & tail = cells[j].tail;
if (tail == src_id) {
tail = dst_id;
} else if (tail == dst_id) {
tail = src_id;
}
}When evaluating a context window of 4096 tokens, this loop performed over 4,000 checks per decoded token on the host CPU thread. This
Since the runtime already knows the exact sequence IDs whose cells are being swapped (dst_cell.seq_id and src_cell.seq_id), we refactored this to directly update the sequence tails. This slashes the complexity to
// OPTIMIZED: O(active_sequences) per token (practically O(1))
for (llama_seq_id seq_id : dst_cell.seq_id) {
cells[seq_id].tail = dst_id;
}
for (llama_seq_id seq_id : src_cell.seq_id) {
cells[seq_id].tail = src_id;
}This direct-pointer update completely eliminates the context-size scanning overhead on every decode step.
Whenever a transition between a user prompt and an assistant turn occurs, set_sampler is invoked.
Previously, set_sampler unconditionally flagged sched_need_reserve = true and reinitialized the backend buffers:
bool llama_context::set_sampler(llama_seq_id seq_id, llama_sampler * sampler) {
// ...
sched_need_reserve = true; // <-- Unconditional re-reservation trigger
// ...
}This flag forced a worst-case graph reallocation (sched_reserve()) on the very next decode step. In a multi-turn conversation, this caused a costly memory reservation pause at the beginning of Turn 2, which disrupted the pipeline, de-allocated NPU weight-resident caches, and caused empty outputs or massive initial token generation pauses.
We added a short-circuit early-return pointer comparison. If the sampler being set is already identical to the active sampler for that sequence, set_sampler exits immediately without triggering the expensive re-reservation:
bool llama_context::set_sampler(llama_seq_id seq_id, llama_sampler * sampler) {
if (!sampler && sampling.samplers.count(seq_id) == 0) {
return true;
}
if (sampler && sampling.samplers.count(seq_id) > 0 && sampling.samplers[seq_id] == sampler) {
return true; // <-- Skip redundant reinitialization and re-reservation
}
// ...
}This prevents redundant buffer reallocations, ensuring that warm model structures and NPU drivers remain fully operational and uninterrupted across turn boundaries.
The optimizations were verified on a Radxa ROCK 5B SBC (RK3588, 16GB LPDDR4X) using the bench-two-turn.mjs test harness with prefix caching enabled.
- Model Type: Standard Dense Transformer
- Quantization: Hybrid (W4A4 Feed-Forward, W8A8 Attention)
| Turn | Turn Type | Prefill Throughput | Decode Throughput | TTFT (ms) |
|---|---|---|---|---|
| Turn 1 | User Turn 1 | 38.3 t/s | 17.9 t/s | 5667 ms |
| Turn 2 | Chat Continuation | 17.6 t/s | 17.8 t/s | 5475 ms |
- Turn 1 vs. Turn 2 Decode Ratio: 1.01× (Complete performance parity; zero slowdown).
- Model Type: Recurrent/Hybrid Attention-Linear Architecture
- Quantization: Hybrid (W4A4 Feed-Forward, W8A8 Attention)
| Turn | Turn Type | Prefill Throughput | Decode Throughput | TTFT (ms) |
|---|---|---|---|---|
| Turn 1 | User Turn 1 | 29.5 t/s | 19.0 t/s | 18045 ms |
| Turn 2 | Chat Continuation | 31.4 t/s | 18.5 t/s | 5631 ms |
- Turn 1 vs. Turn 2 Decode Ratio: 1.03× (Complete performance parity; zero slowdown).
The flow of token execution across turns is now highly streamlined, ensuring that the host CPU threads never starve the NPU:
graph TD
A[Start Chat Turn] --> B{Sampler Changed?}
B -- No --> C[Bypass set_sampler Re-Reservation]
B -- Yes --> D[Flag sched_need_reserve & Reallocate]
C --> E[Assemble Decode Batch]
D --> E
E --> F{Recurrent Architecture?}
F -- Yes --> G[Perform O(1) Tail Swap on Active Sequences]
F -- No --> H[Skip Tail Swap]
G --> I[Route Decode to CPU NEON / Prefill to NPU]
H --> I
I --> J[Generate Output Token]
By shifting the recurrent state sequence tail-swapping from a costly
These enhancements guarantee that llama.cpp + ggml-ork maintains stable, liquid-smooth token generation speeds across long chat conversations, fully unlocking the potential of Rockchip RK3588 hardware.