Async Re-Evaluation #43
Description
Description
When using an LLM-judge similarity evaluator, vCache needs to make this evaluation asynchronous. Otherwise, the end-to-end latency will blow up.
Requirements
- Followed coding conventions
- Implemented or updated tests
Impact
- Who: Users
- What: Async re-evaluation
- Why: Latency
Proposed Solution
vCache should allow the user to enable async re-evaluation. When enabled, a given similarity evaluator should be executed async. The challenge lies in updating the vector database and metadata accordingly. Please draft an outline and discuss it with the vCache team before implementing it. The implementation should be loosely coupled to avoid spaghetti code or other anti-patterns.
Acceptance Criteria
- Outline (doc, uml, etc.)
- Verified by vCache team
- Loosely coupled async implementation
Risks & Dependencies
Wrong async logic leading to incorrect vector database and metadata updates (i.e., race conditions).
Additional Context
--