[Case Study] Agent Context Compaction Breaks RadixAttention Prefix Caching (10.1% -> 99.6% Hit Rate)

[yansu202511]

Hi SGLang team & @LianminZheng,

We wanted to share a production case study that directly validates and highlights the sensitivity of RadixAttention / Prefix Caching in long-horizon LLM Agent workflows.

During multi-hour agent runs, we observed that context compaction (summarization) caused a massive prefill stall (TTFT >200s on ~80k context), collapsing the prefix cache hit rate to only 10.1%.

---

The Mechanism (Breaking the Radix Tree)

The root cause was an upper-layer orchestration mistake:

1. When triggering compaction, the agent stripped the previous summary checkpoint from the messages list to "save tokens".
2. This shifted all subsequent messages forward, causing the serialized request to diverge from the cached sequence at the very first content message (token ~8,886).
3. The Radix tree had to branch right after the system header + tool definitions, forcing a full 73,000-token re-prefill.

The Fix

By simply keeping the compaction request's message prefix token-identical with normal conversation turns (and extracting the archive for output splicing without deleting it from the prompt), the cache hit rate immediately recovered from 10.1% to 99.6%:

• Cached Prefix Reused: 8,192 tok -> 144,165 tok
• Re-prefilled Tokens: 72,986 tok -> 627 tok (116x reduction)
• TTFT: 206.9s -> 3.1s (66x faster)

---

Full Repro Kit & D