Customize prefix caching

[thiomajid]

Hi everyone, I have been tinkering with LLM serving lately and I was wondering if it was possible to provide a custom implementation for prefix caching when serving a model in the mode that accepts input embeddings rather than token ids list.

I know that floating point arithmetic is tricky reason why prefix caching is used for integer tokens alone, but let's say that I'd like to try something custom that when the endpoint receives direct input embeddings, is there a way to do so or I need to fork the engine?

As per the docs, I couldn't find a plugin mechanism to do so.

[QNan0513]

SGLang's prefix caching (RadixAttention) uses integer token IDs​ as keys for its radix tree, not float vectors. If your endpoint receives raw float embeddings instead of token IDs, there is no path in the engine to match those against the radix tree.

[KushagraKanaujia]

Hey, building Throttle: semantic cache for self-hosted inference. Cuts repeat-query costs 40-60%, zero code changes. Works with vLLM, Ollama, SGLang, LMDeploy.

Open-source, free beta. Looking for users to test:

https://github.com/KushagraKanaujia/throttle