Skip to content

v0.1.0 - Initial Release

Choose a tag to compare

View all tags
@Keyvanhardani Keyvanhardani released this 13 Jan 14:25
· 16 commits to master since this release
v0.1.0
68f7ad7

KVCache Auto-Tuner v0.1.0

Automatic KV-Cache Optimization for HuggingFace Transformers

Features

  • Automatic optimization of cache strategy, attention backend, and dtype
  • Built-in profiles: chat-agent, rag, longform, ci-micro
  • CLI interface: kvat tune, kvat apply, kvat compare
  • Production-ready code snippets and JSON configs
  • Markdown & HTML benchmark reports
  • CUDA/GPU memory tracking
  • Windows & Linux support

Installation

pip install kvat[full]

Quick Start

kvat tune gpt2 --profile ci-micro
kvat tune meta-llama/Llama-3.2-1B --profile chat-agent

Benchmarks

  • GPT-2: 124.6 tok/s (RTX 4060)
  • Server: 365 tok/s (RTX 4000 Ada)

Made from Germany with dedication for the HuggingFace community

Assets 2
Loading