Skip to content

v0.2

Choose a tag to compare

View all tags
@sysit sysit released this 25 Mar 16:42
· 36 commits to main since this release
v0.2
b6dad2b

MLX Service v0.2

New Features

  • Prefix Cache: KV cache reuse with LRU eviction
    • Token hash-based cache key
    • Configurable memory limits
    • Cache hit/miss statistics

New Endpoints

  • GET /v1/cache/stats - View cache statistics
  • POST /v1/cache/clear - Clear cache

Fixes (from v0.1)

  • Ollama API proper error status codes (400/503)
  • Auth whitelist for read-only endpoints

Tested

  • ✅ Cache hit rate tracking
  • ✅ Sync and stream generation
  • ✅ ~22 tokens saved per cache hit
Assets 2
Loading