Skip to content

[Feature] Optimize session cache performance #37

Description

@b-re-w

Issue: No session‑level KV cache leads to redundant re-processing of the full chat history on every request, degrading performance

Implement session-scoped KV cache support

  • Persist cache state per session:
    Add attributes (e.g., kv_cache, past_key_values) to Session in api/src/main/settings.py to hold per-session model state. Provide methods to retrieve, update, and clear these caches when a session ends.
  • Integrate cache with BaseModel/runtime calls:
    Extend BaseModel.chat and CoreRuntime.__call__ to accept optional cache objects from the Session and return updated cache data after generation.
  • Clear cache on session close:
    Modify Session.close to explicitly discard any stored cache state and free related resources.

Add KV cache utilization to llama_cpp backend

  • Store and reuse llama.cpp state:
    In api/src/main/backend/gguf.py’s GGUFRuntime.__call__, leverage save_session_file/load_session_file (or similar llama_cpp APIs) to persist key/value caches between calls within the same session.
  • Expose cache hooks:
    Accept an incoming cache handle and return the updated handle so Session can manage it. Ensure caches are reset when Session.clean_up or Session.close is invoked.

Enable past key/value caching for Transformers backend

  • Manage past_key_values:
    In api/src/main/backend/bin.py, maintain past_key_values from Hugging Face model outputs. Feed this cache back on subsequent calls to avoid re-tokenizing the entire history.
  • Session integration:
    Pass past_key_values to/from the Session cache helpers. Reset them on session termination or when the conversation is cleared.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions