You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
perf: aggressive KV cache optimizations
- Pre-allocate messages_for_api Vec with exact capacity to avoid realloc
- Cache serialized tool definitions in OpenAIProvider (serialize once at
construction, clone for each request instead of rebuilding)
- Add cache_control {"type": "ephemeral"} annotation to system
ChatMessage so compatible servers (vLLM, llama.cpp, etc.) explicitly
cache the KV state for the system prompt and all preceding messages
- Cap tool results at 4000 chars when storing in conversation history,
keeping first ~70% + last ~30% with truncation marker. Full results
still go to the event stream; only the stored copy is bounded.
This reduces per-turn prefill work by preventing huge file reads
from bloating every subsequent KV cache computation.