KV cache in server mode?

[artenais]

Good day,

Apparently from my understanding KV cache works only in CLI mode? Is there any plan to implement this in the future, like llama.cpp does?

Thank you for your attention.

[ZaneNi]

Good day,

Apparently from my understanding KV cache works only in CLI mode? Is there any plan to implement this in the future, like llama.cpp does?

Thank you for your attention.

@artenais We have KV cache which is supported in both CLI and server modes.

You can refer to this video for more details if you’re interested.

[artenais]

@ZaneNi I may be cofused by :

⚠️ The OpenAI API (and FastFlowLM server mode) is stateless — you must resend the full conversation each time. No KV cache is kept between turns.

FastFlowLM’s CLI mode uses a real KV cache, making multi-turn responses much faster — especially with long conversations.

I will do some test in the coming days as soon I as I will be able.

[artenais]

I run some test and this does not seem to be the case with Ubuntu 25.x :

LOG] Target: /v1/chat/completions
[LOG] Version: 11
[LOG] Keep-Alive: 1
[LOG] Body:
{
"cache_prompt": true, < this should enable caching like in llama.cpp
"dry_allowed_length": 2,
"dry_base": 1.75,
"dry_multiplier": 0,
"dry_penalty_last_n": -1,
"dynatemp_exponent": 1,
"dynatemp_range": 0,
"frequency_penalty": 0,
"max_tokens": 87,
"messages": [
{
"content": "You are a ..._MESSAGE}}",
"role": "system"
},
{
"content": "how is your day",
"role": "user"
},
{
"content": "I'm just a...he agenda?",
"role": "assistant"
},
{
"content": "ok now how...e you doin",
"role": "user"
}
],
"min_p": 0.05,
"presence_penalty": 0,
"repeat_last_n": 64,
"repeat_penalty": 1.1,
"samplers": "edkypmxt",
"stream": false,
"temperature": 0.8,
"timings_per_token": true,
"top_k": 40,
"top_p": 0.95,
"typical_p": 1,
"xtc_probability": 0,
"xtc_threshold": 0.1
} > the response times get bigger and bigger, making it evident that the full chat prompt is being reprocessed and not just the last messages. I am using Phi4.
CLI seems to hit KV cache consistently so no problem.
I am surely making something wrong here, can you be so kind to give me any hint?

[ZaneNi]

I run some test and this does not seem to be the case with Ubuntu 25.x :

LOG] Target: /v1/chat/completions [LOG] Version: 11 [LOG] Keep-Alive: 1 [LOG] Body: { "cache_prompt": true, < this should enable caching like in llama.cpp "dry_allowed_length": 2, "dry_base": 1.75, "dry_multiplier": 0, "dry_penalty_last_n": -1, "dynatemp_exponent": 1, "dynatemp_range": 0, "frequency_penalty": 0, "max_tokens": 87, "messages": [ { "content": "You are a ..._MESSAGE}}", "role": "system" }, { "content": "how is your day", "role": "user" }, { "content": "I'm just a...he agenda?", "role": "assistant" }, { "content": "ok now how...e you doin", "role": "user" } ], "min_p": 0.05, "presence_penalty": 0, "repeat_last_n": 64, "repeat_penalty": 1.1, "samplers": "edkypmxt", "stream": false, "temperature": 0.8, "timings_per_token": true, "top_k": 40, "top_p": 0.95, "typical_p": 1, "xtc_probability": 0, "xtc_threshold": 0.1 } > the response times get bigger and bigger, making it evident that the full chat prompt is being reprocessed and not just the last messages. I am using Phi4. CLI seems to hit KV cache consistently so no problem. I am surely making something wrong here, can you be so kind to give me any hint?

Could u provide more information so that I could reproduce it? Thank you!

[thunderbug1]

I also observe this, it can be easily tested by using "flm serve" and then sending it a very large system prompt before the user message ( I hooked up claude-caude, which has a beefy 112301 byte system prompt, just to see if it would work ).

The first message I sent it a "HI" and it took 2:20min for the answer, the second message I sent was "Cool" and it took 2:29min to get an answer so it looks like the whole system prompt was processed again since the user message was tiny.

[thunderbug1]

if you are using claude-code you might encounter this issue: https://www.reddit.com/r/LocalLLaMA/comments/1s7tn5s/psa_using_claude_code_without_anthropic_how_to/#:~:text=You%20can%20fix%20this%20in,is%20touched%2C%20the%20prompt%20changes.

[artenais]

@thunderbug1 Thank you for bringing this to notice. I was unable to do further tests due to overwork :)

[mariolpantunes]

KV cache was supposed to also be used in server mode? I tried goose with flm, but is unusable due to the long pre fill times...

[artenais]

mmm I suppose the fix is still not merged because in .43 build I have the same behaviour. Did I miss something?

[ZaneNi]

mmm I suppose the fix is still not merged because in .43 build I have the same behaviour. Did I miss something?

@artenais May I ask whether you are using non-streaming mode? I apologize that the KV cache is not supported in non-streaming mode yet.

However, if you are using streaming mode, KV cache should work unless something in the conversation history has changed, such as the system prompt or tools.

[artenais]

This may sound stupid but I am using FastFlow as a classifier. like focus : "chatting". It makes just sense to me to avoid streaming ?

[ZaneNi]

This may sound stupid but I am using FastFlow as a classifier. like focus : "chatting". It makes just sense to me to avoid streaming ?

Not stupid at all. It's interesting and also makes sense to me. We will support non-stream kv cache in the next release. Please stay tuned.

[artenais]

the .44 not merged it correct?

[artenais]

0001-fix-invalidate-prompt-cache-alongside-KV-context-on-.patch

0001-feat-honor-cache_prompt-false-in-v1-chat-completions.patch

After some analyzing of the problem, the previous patch failes on .45 because of changed implementation about cache-hopping NPU <> GPU. This should re-enable the prompt caching in non-streaming mode.