Reduce local-attention KV storage with per-layer rings - #1017
Closed
Mikyx-1 wants to merge 1 commit into
Closed
Conversation
Mikyx-1
force-pushed
the
kv-cache-local-windows
branch
from
September 6, 2026 14:16
a80493f to
56bf38a
Compare
Mikyx-1
marked this pull request as draft
September 6, 2026 16:42
Author
|
Superseded by #1020, which ports the local-cache changes to dev and includes new dev-baseline validation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1016
Local-attention layers currently allocate KV storage for the entire context. This change gives each layer its own FP32 buffer: local layers use
min(seq_len, window - 1 + max(1, prefill_batch_size))rows, and global layers keep full-context storage.The extra batch rows prevent QKV prefill writes from overwriting history needed by earlier queries. Runtime growth preserves cached history, copies remain independent, and attention reads use each layer's ring capacity while score-buffer indexing and arithmetic remain unchanged.
Validation on Gemma 3 270M and 1B: all 10 A/B comparisons produced byte-identical decoded output and identical token IDs (1,280 generated tokens per build). Runs used a shared baseline-calibrated matrix plan to isolate the cache change from timing-dependent autotuning.
Medians from three alternating pairs with a 3,367-token prompt, 128 generated tokens, 8,192-token capacity, 256-token prefill batches, and six pinned workers on an i5-12400F. CPU load varied; speed improvements are measured observations, not a universal performance guarantee. RAM savings were consistent.
C++ API change: direct users of the former public
KVCache::kv_cachematrix must useLayerCache(layer)orRow(layer, position). Repository callers are updated.