Repository navigation
Replies: 2 comments 1 reply
|
@markg85 It is unlikely FP will be zero. But at the moment, we have not yet evaluated the cache hit in a production workload with different threshold configurations. It is likely a good idea to create such benchmark/dataset to conduct the cache hit eval first. |
|
I would not treat this as a general chat-history memory. It is safer to think of it as route-local response reuse. For the C++ build example, the risky part is missing state in the cache identity. The question "how do I build this repo?" can have a different correct answer depending on repo path, branch/commit, build directory, CMake generator, toolchain, and even the current The mitigations I would use are:
So I would not try to "fix" an old hit by another similar question. I would treat cached responses as derived artifacts: when the underlying state changes, invalidate that scope or write under a new fingerprint. On the context-window question: yes, the cache can outlive the model context if the backend and TTL allow it and the compatibility fingerprint still matches. But it only preserves previous responses. It does not know the current build directory or project configuration unless that state is part of what gets cached or partitioned. For evaluation, I would build a small hit/miss set before choosing a threshold:
That gives you a concrete way to decide whether the route is safe enough for build guidance instead of relying on a single similarity number. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi,
Some disclaimers first.
Congrats on this project! It looks very promising and i want to try it out. I watched a video about it's architecture and if I'm not mistaken then it's essentially a chat history in a rag pipeline, correct? By which i mean that queries are cached and semantic similar queries are send back without ever even hitting the model. RAG might be a bit of an overstatement here as there is no "augmentation" of a query, just a cache hit + return. But it's the same technique just without sending it to the LLM after adding the hit.
With that out of the way i would like to understand how a false positive can be mitigated?
Here's the scenario (and this happens a lot in longer coding sessions. My specific case is a C++ project. The model (name any good coding model, GLM 5, Kimi K2.5, Nemotron 3 super, Qwen Coder, etc...) has identified the project structure and is able to compile it. The problem: There are multiple ways to compile and it, obviously, picks a different way every time.. Like at first it might detect that it's a
cmakeproject and uses ninja to configure/build the whole thing. And then at a later point it cheerfully assumes it's a Makefile based project (which is then already configured as ninja) and will then fail. Causing it to figure out why it got stuck and often picking the route of removing the build folder and reconfiguring it with whatever it things at that time is the best way.The problem i'm identifying here is the ambiguous detection of how a project can be build. One time it make do
ninjato build, another time it might domaketo build. Now in a chat cache (hence the false positive) it might have a answer that it should useninjawhen it in fact was configured as makefile project and it thus should've usedmakeinstead. How do you update the cache hit to accurate again?Also, and this is an aside but related question. Am i correct in assuming that this semantic router project effectively outlives the model context window. So if i clear the context (in the coding agent) and then ask it to compile this project that it still knows (through the cache hit) that it can compile and how it can do that?
And one for the tips category. I'm assuming you folks are up to date on the latest AI research? But in case you missed it, you might want to see this paper from MIT about, effectively, lossless kv compation while maintaining throughput https://arxiv.org/pdf/2602.16284 and here is there github repo https://github.com/adamzweiger/compaction
All reactions