[Kernel Feature] KV-Cache Manager - User-Space Cache Management for LLM Inference

[mikejmorgan-ai]

KV-Cache Manager - User-Space Cache Management for LLM Inference

Part of: Kernel-Level AI Enhancements (Tier 1 - User-Space)

Description

User-space implementation of kernel-level KV-cache concepts. Manages transformer key-value caches as first-class system resources.

Effort: 3-4 weeks | Bounty: $175

The Solution

cortex cache create llama-cache --size 16G --tier cpu
cortex cache status llama-cache
cortex cache persist llama-cache  # save to disk
cortex cache restore llama-cache  # fast restart
cortex cache evict llama-cache --percent 25

Features

• POSIX shared memory pools
• Cross-process cache sharing
• Multiple eviction policies (LRU, LFU, FIFO, priority)
• Prefix hash matching for cache sharing
• Disk persistence for fast restarts
• Thread-safe allocator with bitmap free list

Memory Layout

┌──────────────────┐
│ Header (4KB)     │ Magic, version, usage stats
├──────────────────┤
│ Free List (4KB)  │ Bitmap of free blocks
├──────────────────┤
│ Data Region      │ KV-cache tensors
└──────────────────┘

Acceptance Criteria

• Cache pools can be created and destroyed
• Multiple processes can attach to same pool
• Eviction works correctly under pressure
• Persistence and restore works
• Unit tests pass with >80% coverage

Patent Connection

This implements user-space versions of concepts in our provisional patent for kernel-managed KV-cache memory regions.

Files

Complete implementation available:

• kv_cache_manager.py (~900 lines)
• test_kv_cache.py (~400 lines)
• README_KV_CACHE.md

Priority

High - Core infrastructure for AI-native OS

[rakesh0x]

hey @mikejmorgan-ai i want to work on this pr, would you please assign it to me ?

[yaya1738]

Interested in this bounty.

Relevant experience:

• POSIX shared memory and mmap for cross-process communication
• LRU/LFU cache implementations
• Python systems programming (ctypes, multiprocessing)
• Built production autonomous AI systems

Approach:

• POSIX shm_open + mmap for shared memory pools
• Thread-safe allocator with bitmap free list
• Multiple eviction policies (LRU, LFU, FIFO, priority-based)
• Prefix hash matching for cache sharing across requests
• SQLite or memory-mapped file for persistence

Key components:

1. CachePool - shared memory region management
2. CacheAllocator - thread-safe block allocation
3. EvictionManager - policy-based eviction
4. PersistenceManager - disk persistence/restore
5. CLI - cortex cache create/status/persist/restore/evict

Can start immediately alongside #222 and #223 (already have working prototypes for those).

[yaya1738]

KV-Cache Manager Prototype Complete

I've built a full working prototype for the KV-cache manager targeting LLM inference workloads.

Features Implemented

Core Components:

• KVCachePool - POSIX shared memory pools with mmap-based storage
• BitmapAllocator - First-fit block allocator with bitmap-based free list
• EvictionManager - Pluggable eviction policies (LRU, LFU, FIFO, Priority)
• CacheStore - Multi-pool management with persistence

Memory Layout:

┌──────────────────────────────┐
│ Header Block (4KB)           │ - Magic, version, entry count
├──────────────────────────────┤
│ Allocation Bitmap (4KB)      │ - Free/used block tracking
├──────────────────────────────┤
│ Data Region                  │ - Actual KV cache storage
│ (size - 8KB)                 │
└──────────────────────────────┘

Cache Operations:

• put(key, data) - Store KV cache with prefix hash for sharing
• get(key) - Retrieve with access tracking
• delete(key) - Free blocks and update metadata
• evict(count) - Policy-based eviction
• find_by_prefix(hash) - Find caches sharing prompt prefix

Eviction Policies:

• LRU (Least Recently Used) - Based on last_accessed timestamp
• LFU (Least Frequently Used) - Based on access_count
• FIFO (First In First Out) - Based on created_at
• Priority - Based on user-defined priority scores

CLI Interface:

cortex cache create llama-kv --size 16G --tier gpu --policy lru
cortex cache status llama-kv
cortex cache put llama-kv --key "prompt1" --file kv_tensor.bin
cortex cache get llama-kv --key "prompt1"
cortex cache evict llama-kv --percent 25
cortex cache persist llama-kv --output /data/cache.dat
cortex cache restore /data/cache.dat

Persistence:

• Full pool serialization (header + bitmap + entries + data)
• Atomic restore with validation
• Cross-session cache continuity

Test Results

49 passed in 0.13s

Test coverage includes:

• Size parsing/formatting utilities
• CacheEntry serialization
• BitmapAllocator allocation/free/serialization
• All 4 eviction policies
• KVCachePool put/get/delete/evict/stats
• Pool persistence and restore
• CacheStore multi-pool management
• End-to-end LLM workflow simulation

Implementation Stats

• ~900 lines of Python
• Zero external dependencies beyond stdlib
• Thread-safe operations with proper locking
• Designed for integration with vLLM/TGI continuous batching

Ready to share the full implementation for review.