Declarative Attention (DA): a KV-cache "hot window + hole punching" technique for long-context speed in llama.cpp — focus-llama

[edwardyoon]

I've been running Qwen3.8-27B (Q6_K quant) on a single RTX 5090 for coding work, and wanted to keep long-context sessions fast without giving up context length. Inspired by the Declarative Attention (DA) concept from a recent Google DeepMind paper, I implemented it as a patch on top of llama.cpp — calling the resulting project focus-llama.

What it does:

• Keeps a full 230k token context available, but only maintains a 3~40k "hot" window of KV cache actively attended over.
• Tokens outside the hot window aren't dropped — they're offloaded, punching a "hole" in the KV cache, and refilled/reattached on demand when attention needs to reach back into them.
• Net effect: attention stays focused on a much smaller working set most of the time, so per-token cost tracks the hot window size, not the full context length.

Benchmark (1hr identical coding session, RTX 5090, Q6_K, 230K context length):

	Vanilla llama.cpp	focus-llama (DA)
Avg TPS	76.2 t/s	102.7 t/s

~35% throughput improvement in this run, without truncating or summarizing context — the full 230k window is still logically there, just not all "hot" at once.

KV offload here does not reduce actual usable context. It's a hole, not a cut — anything offloaded can be refilled and attended to again.

Note that the sparse gather kernel currently supports f16 and q4_0 KV cache only. Other KV quant types fall back to the dense path automatically (no error, just no speed-up for that request).

Roadmap: planning to move to paged KV blocks, at which point the current hole-punching / auto-compaction mechanism becomes unnecessary — paging handles the same problem more generally.

link: https://github.com/edwardyoon/focus-llama

[159753a52]

Nice result. Two memory-side details would make the comparison easier to read.

Which KV type the vanilla run used.

• A Q6_K file of this model is about 20.5 GiB (unsloth's UD-Q6_K is 21,983,677,344 bytes).
• Its KV cache at 230K (235,520 tokens) is 14.4 GiB in f16, 7.6 GiB in q8_0 and 4.0 GiB in q4_0: 16 of the 64 layers keep KV, 64 KiB per token in f16.
• So f16 at 230K is ~34.9 GiB before compute buffers and can't fit a 32 GB card. The runs must be q8_0 or q4_0.
• Since the sparse gather only handles f16 and q4_0, it would help to state that both runs used q4_0. If vanilla was on q8_0, the comparison also changes the KV type.

Whether speculative decoding was on.

• A 5090 has 1,792 GB/s, and a ~22 GB Q6_K is read once per token, so plain decode tops out around 78–81 t/s even at short context, and ~66 t/s at 230K with a q4_0 cache.
• 76.2 t/s averaged over an hour of coding is at that ceiling, which suggests MTP (or another draft) is on in both runs.
• Worth saying, because draft acceptance changes with what the session is doing, so two one-hour sessions can differ on that alone.

On the attention side: at full context a q4_0 cache is 4.3 GB read per decoded token, against 0.75 GB for a 40K hot window. Of the ~26 GB read per token, that's a saving of about 3.6 GB, i.e. at most ~15% from bandwidth alone at 230K, and less while the context is still short. A 35% average gain therefore points at something beyond decode bandwidth, e.g. cheaper verify batches for the drafts, which would be interesting to break out.

Estimates, not measured on a 5090. The KV and bandwidth figures are from a calculator I built: [Link removed by moderator]/llm-vram-calculator/qwen3.8-27b/

[edwardyoon]

Here's my setup:

ExecStart=/home/edwardyoon/focus-llama/build/bin/llama-server
-m /home/edwardyoon/my_model/qwen3.8/Qwen3.8-27B-AD-Q6_K.gguf
--mmproj /home/edwardyoon/my_model/qwen3.8/mmproj-BF16.gguf
--no-mmproj-offload
--alias qwen27b
-ngl 99
-b 1024
-ub 1024
-c 210000
--host 0.0.0.0
--port 8080
--numa numactl
--jinja
--cont-batching
--timeout 600
--parallel 1
--flash-attn on
--cache-reuse 128
--cache-type-k q4_0
--cache-type-v q4_0
--no-webui
--metrics
--no-context-shift
--da-auto
--kv-unified
--da-min-ctx 2048
--da-chunk-tokens 4096
--fm-offload
--kv-offload-threshold 36865
--kv-offload-holes
--sparse-gate-threshold 60
--focus-memory-host http://127.0.0.1:3900
--focus-memory-token x
--spec-type draft-mtp
--spec-draft-n-max 4
--spec-draft-ngl all
--spec-draft-type-k q8_0
--spec-draft-type-v q8_0
--cache-ram 22000
--temperature 0.9
--top-k 30
--top-p 0.90
--min-p 0.0
--presence_penalty 0.0
--repeat-penalty 1.0
--repeat-last-n 64
--reasoning on
--no-reasoning-preserve