You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Issue: No session‑level KV cache leads to redundant re-processing of the full chat history on every request, degrading performance
Implement session-scoped KV cache support
Persist cache state per session:
Add attributes (e.g., kv_cache, past_key_values) to Session in api/src/main/settings.py to hold per-session model state. Provide methods to retrieve, update, and clear these caches when a session ends.
Integrate cache with BaseModel/runtime calls:
Extend BaseModel.chat and CoreRuntime.__call__ to accept optional cache objects from the Session and return updated cache data after generation.
Clear cache on session close:
Modify Session.close to explicitly discard any stored cache state and free related resources.
Add KV cache utilization to llama_cpp backend
Store and reuse llama.cpp state:
In api/src/main/backend/gguf.py’s GGUFRuntime.__call__, leverage save_session_file/load_session_file (or similar llama_cpp APIs) to persist key/value caches between calls within the same session.
Expose cache hooks:
Accept an incoming cache handle and return the updated handle so Session can manage it. Ensure caches are reset when Session.clean_up or Session.close is invoked.
Enable past key/value caching for Transformers backend
Manage past_key_values:
In api/src/main/backend/bin.py, maintain past_key_values from Hugging Face model outputs. Feed this cache back on subsequent calls to avoid re-tokenizing the entire history.
Session integration:
Pass past_key_values to/from the Session cache helpers. Reset them on session termination or when the conversation is cleared.
Issue: No session‑level KV cache leads to redundant re-processing of the full chat history on every request, degrading performance
Implement session-scoped KV cache support
Add attributes (e.g.,
kv_cache,past_key_values) toSessioninapi/src/main/settings.pyto hold per-session model state. Provide methods to retrieve, update, and clear these caches when a session ends.BaseModel/runtime calls:Extend
BaseModel.chatandCoreRuntime.__call__to accept optional cache objects from theSessionand return updated cache data after generation.Modify
Session.closeto explicitly discard any stored cache state and free related resources.Add KV cache utilization to llama_cpp backend
In
api/src/main/backend/gguf.py’sGGUFRuntime.__call__, leveragesave_session_file/load_session_file(or similar llama_cpp APIs) to persist key/value caches between calls within the same session.Accept an incoming cache handle and return the updated handle so
Sessioncan manage it. Ensure caches are reset whenSession.clean_uporSession.closeis invoked.Enable past key/value caching for Transformers backend
past_key_values:In
api/src/main/backend/bin.py, maintainpast_key_valuesfrom Hugging Face model outputs. Feed this cache back on subsequent calls to avoid re-tokenizing the entire history.Pass
past_key_valuesto/from theSessioncache helpers. Reset them on session termination or when the conversation is cleared.