Repository navigation
[RFC] Semantic cache entry identity: should the key be query-only, or composite (tenant × model × query)? #2180
theohsiung
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Context
This is a follow-up to #1593 ("How do you get rid of a false positive cache hit?"), where @rootfs noted that FP is unlikely to be zero and that we have not yet evaluated cache hits in a production workload with different thresholds — and that a cache-hit benchmark/dataset would be a good first step.
While reviewing the cache subsystem I found that several of the false-positive classes don't come from the threshold at all — they come from what we treat as a cache entry's identity. Today the de-facto key is essentially
embedding(query)+ a similarity threshold, and two other dimensions that intuitively should scope a hit are either ignored or only softly separated. I'd like to open a design discussion on this before proposing code, because the answer is a product decision, not a bug fix.Current behavior (verified on
main)modelis accepted but never used as a filter.FindSimilar(model, query)/FindSimilarWithThreshold(model, query, threshold)carrymodel, and every backend storesentry.Model, but no backend uses it in the search/ranking path (in-memory linear/HNSW scan, Redis/Valkey KNN, Milvus expr). So a query routed to model A can be served a response that was generated by model B, as long as the query embeddings are similar.Multi-tenant isolation is soft only (and structural only for in-memory).
ScopeQueryToUser(query, userID)prepends an HMAC namespace token to the query text so different users' vectors tend to land apart in embedding space. That separation is statistical, not structural. [Router] Semantic cache user-scoping is soft (embedding-distance) → cross-tenant response leak (CWE-524) #2134 reproduced one user receiving another user's cached response at cosine sim 0.9085. [Router] Enforce hard user-scope isolation in semantic cache (cross-tenant leak fix) #2135 adds a hard namespace gate — but only in the in-memory backend; Redis/Valkey/Milvus/Qdrant still rely on the soft prefix.similarity_threshold == 0.0passes validation (the factory only rejects< 0or> 1), and0.0is the float32 zero value — an operator who forgets to set it gets match-all.The design question
What dimensions should compose a semantic cache entry's identity?
Right now it's effectively
(query-embedding). The two candidates that keep coming up aremodelandtenant. Three broad options:queryonly(tenant, model, query)scope: global | per-model | per-tenant | per-model-per-tenantIn B/C the
tenantandmodelpredicates would be exact-match pre-filters applied before similarity ranking (Redis/Valkey TAG, Qdrantmustfilter, Milvus boolean expr, in-memory field compare) —querystays semantic.What this does not solve
The #1593 scenario (the model answered "build with
ninja" but the project was later reconfigured formake) is a staleness / external-state problem: even a perfect(tenant, model, query)key returns a stale-but-correct-by-key hit, because the right answer depends on mutable state outside the query. That's a separate axis (TTL / invalidation / freshness), and I think it's exactly the kind of thing rootfs's proposed cache-hit eval would quantify. I'm scoping this RFC to the key/scope dimensions only.Questions for discussion
CacheBackendseam, or per-backend implementations?Happy to prototype whichever direction the maintainers prefer, and to help build the eval harness if that's the agreed first step.
cc @rootfs @Xunzhuo (router-runtime owners) — would value your steer here, especially on Q1 (is cross-model reuse intended?) and Q2 (should hard tenant isolation be the default?). Building on @rootfs's note in #1593 about needing a cache-hit eval.
All reactions