Skip to content

v0.7.24

Choose a tag to compare

@github-actions github-actions released this 21 Jul 19:04
· 21 commits to main since this release
perf: aggressive KV cache optimizations

- Pre-allocate messages_for_api Vec with exact capacity to avoid realloc
- Cache serialized tool definitions in OpenAIProvider (serialize once at
  construction, clone for each request instead of rebuilding)
- Add cache_control {"type": "ephemeral"} annotation to system
  ChatMessage so compatible servers (vLLM, llama.cpp, etc.) explicitly
  cache the KV state for the system prompt and all preceding messages
- Cap tool results at 4000 chars when storing in conversation history,
  keeping first ~70% + last ~30% with truncation marker. Full results
  still go to the event stream; only the stored copy is bounded.
  This reduces per-turn prefill work by preventing huge file reads
  from bloating every subsequent KV cache computation.