Is there a way to cache multiple prompt prefixes?

[ghnp5]

Hi,

I'm using the OpenAI-compatible API, running GGUF on a CPU, with the llama.cpp loader.

--streaming-llm is very useful to cache the last prompt prefix, so that the next time it runs, it will have to process the prompt only from the first token that is different.

However, in my case, I will have about 8 prompt prefixes that will be rotating all the time. This makes --streaming-llm mostly useless.

Is there a way to cache 8 variations of the prompt prefixes? (while still allowing me to inject suffixes that will always be different, and not expected to be cached)

Many thanks!

---

EDIT - I suspect this may be possible, by enabling "slots", and then play with --keep and -sps, --slot-prompt-similarity SIMILARITY. I don't think I can pass "any parameters" to llama-server from CMD_FLAGS.txt, though.

[ghnp5]

This tutorial may show how to do this: ggml-org/llama.cpp#13606

But we'd need oobabooga/text-generation-webui to support proxying these parameters onto llama.cpp.