Skip to content

v0.8.0: Sliding Window Attention + GQA Optimization + Memory-Mapped KV Cache

Choose a tag to compare

View all tags
@quantumnic quantumnic released this 18 Feb 17:21
· 53 commits to main since this release
v0.8.0
3b9228f

Highlights

🪟 Sliding Window Attention

Limit attention to the most recent W tokens with optional sink tokens. Bounds memory and compute for long-context inference while keeping BOS/system prompt visible.

🔗 GQA Optimization

Dedicated grouped-query attention path. Pre-fetches KV vectors once per group, uses 4-wide SIMD dot products. Auto-detects MHA/GQA/MQA from model config. Llama 2 70B gets 75% KV memory savings.

💾 Memory-Mapped KV Cache

Hot/cold architecture: recent KV entries in RAM, older entries spill to SSD via mmap. Enables ultra-long contexts on memory-constrained devices. Automatic spill and OS-managed paging.

Stats

  • 3 new modules, 18 new tests (55 total, all passing)
  • New CLI flags: --sliding-window, --sink-tokens, --mmap-kv

See CHANGELOG.md for full details.

Assets 2
Loading