Eval bug: cache-reuse not effective in qwen3-next

[IMbackK]

Name and Version

llama-server at commit 0f89d2e

Operating systems

Linux

GGML backends

HIP

Hardware

MI100s

Models

Qwen3-Next-80B-A3B-Instruct-Q5_K_S.gguf

Problem description & steps to reproduce

I am not sure if this is a bug or a model incompatibility with Qwen3-Next but using Qwen3-Next kv cache reuse seams to not work.

llama-server is started like this:

llama-server -ngl 99 -fa 1 -m Qwen3-Next-80B-A3B-Instruct-Q5_K_S.gguf --jinja -c 64000 -np 2 -sps 0.3 --cache-reuse 128 -cram 65536 --port 8092 -ub 2048 --swa-full

then a request with a context of 4096 tokens is sent, kv looks like this with each letter being 512 tokens long block:

AAAABBBB

Then a new request is sent with the same content except for an extra block of 512 tokens inserted in the center:

AAAACBBBB

Result with Qwen3-Next:

Context gets fully reporccesed:

slot update_slots: id  3 | task 173 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)

Result with Qwen3:

512 tokens get processed, the rest is filled from shfited kv cache.

I also tried with --ctx-checkpoints 128 but this did not change the behavior, im not sure how this feature is supposed to work.

[pwilkin]

The problem is the recurrent state, not the KV cache. You will get checkpoints after each processed prompt, so if a checkpoint matches, content will get reprocessed from there. But the recurrent state doesn't work like the KV cache. It only keeps a single state that gets updated with each token. Therefore, the only way this could work if we added arbitrary snapshot checkpoints during prompt processing, for example each N tokens.

[othermod]

This seems to be the issue I'm experiencing too. I send 100k tokens of context (which stays the same) + and additional 3k tokens of instruction at the end (which changes on each run), and it always rebuilds the whole thing rather than the last 3k. I switched back to using another model for the task because its taking so long to do the prompt processing on each pass.

[ggerganov]

The --cache-reuse and --swa-flag flags do not do anything for recurrent models, so you can remove them for Qwen3 Next.

If you have control over the client, you can chunk the input prompt in smaller pieces so that the server can create checkpoints and reduce the amount of recomputation for shared prefixes.

Therefore, the only way this could work if we added arbitrary snapshot checkpoints during prompt processing, for example each N tokens.

Yes, this is quite easy to implement. It will likely increase the host-memory cache size, but no way around this. Contributions are welcome.

[othermod]

If you have control over the client, you can chunk the input prompt in smaller pieces so that the server can create checkpoints and reduce the amount of recomputation for shared prefixes.

test_chunking.py

I've tried chunking with a python script and it still no luck. This script adds 5 chunks, followed by an instruction, and the cache works. When the instruction is modified, the cache has to rebuild.

Chunk 1... 3.65s
Chunk 2... 4.18s
Chunk 3... 3.57s
Chunk 4... 4.20s
Chunk 5... 3.51s
First instruction... 0.78s
Modified instruction... 15.60s

This issue only happens on this model.

Here is what I'm seeing in LM Studio when the modified instruction runs. Including in case it helps.
2025-12-31 01:58:00 [INFO]
[qwen3-next-80b-a3b-thinking] Running chat completion on conversation with 11 messages.
2025-12-31 01:58:00 [DEBUG]
Sampling params: repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = -1
top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
2025-12-31 01:58:00 [DEBUG]
Sampling:
logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
Generate: n_ctx = 262144, n_batch = 512, n_predict = 20, n_keep = 805
2025-12-31 01:58:00 [DEBUG]
Looking for non-prefix contiguous prompt sequences of size >= 256 to reuse from cache
Cache reuse summary: 4133/4145 of prompt (99.7105%), 4133 prefix, 0 non-prefix
Failed to remove tokens from cache, clearing cache instead.
2025-12-31 01:58:00 [DEBUG]
Total prompt tokens: 4145
Prompt tokens to decode: 4145
BeginProcessingPrompt

[ggerganov]

Chunking over /v1/chat/completions does not work because the chat templates appends various tokens to each chunks so the prefix will never match.

[IMbackK]

The problem is the recurrent state, not the KV cache. You will get checkpoints after each processed prompt, so if a checkpoint matches, content will get reprocessed from there. But the recurrent state doesn't work like the KV cache. It only keeps a single state that gets updated with each token. Therefore, the only way this could work if we added arbitrary snapshot checkpoints during prompt processing, for example each N tokens.

I see, so the clue I was missing is that qwen3-next has recurrent elements, I just red a paper on deltanet. The behavior makes sense with that in mind. I think better diagnostics would be good when non-applicable flags are used, its a bit mutch to expect the user to know the architecture details of eatch model used.

[vnicolici]

I experience similar issues with LM Studio which currently uses llama.cpp b7721 and qwen3-next-80b. Cache reuse is very random, and it frequently discards the entire cache and recomputes it, which makes using large contexts almost impossible.

For example, during my recent test I used a system prompt 1071 tokens long, followed by 40 tokens for the first user prompt. The initial prompt processing worked as expected when the cache is empty:

Generate: n_ctx = 262144, n_batch = 512, n_predict = -1, n_keep = 0
2026-01-16 14:07:41 [DEBUG]
 Total prompt tokens: 1124
Prompt tokens to decode: 1124
BeginProcessingPrompt
2026-01-16 14:07:48 [DEBUG]
 PromptProcessing: 45.5516
2026-01-16 14:07:53 [DEBUG]
 PromptProcessing: 91.1032
2026-01-16 14:07:57 [DEBUG]
 FinishedProcessingPrompt. Progress: 100

It generated 643 tokens as the assistant reply.

When I tried to regenerate that reply (so the same system prompt and user prompt), I got this:

2026-01-16 14:09:49 [DEBUG]
 Looking for non-prefix contiguous prompt sequences of size >= 256 to reuse from cache
Cache reuse summary: 1124/1124 of prompt (100%), 1124 prefix, 0 non-prefix
Failed to remove tokens from cache, clearing cache instead.

Then after reply regeneration, which produced 2566 tokens, I enter a new 9 token prompt. When generating the next reply, it doesn't match the entire cache, possibly due tokenization issues like those I reported in the past for DeepSeek models. But even the part that matches, the first 1781 tokens out of 3759, can't be used, because the entire cache is discarded again:

2026-01-16 14:33:12 [DEBUG]
 Looking for non-prefix contiguous prompt sequences of size >= 256 to reuse from cache
Cache reuse summary: 1781/3759 of prompt (47.3796%), 1781 prefix, 0 non-prefix
Failed to remove tokens from cache, clearing cache instead.
2026-01-16 14:33:12 [DEBUG]
 Total prompt tokens: 3759
Prompt tokens to decode: 3759
BeginProcessingPrompt
2026-01-16 14:33:18 [DEBUG]
 PromptProcessing: 13.6206
2026-01-16 14:33:24 [DEBUG]
 PromptProcessing: 27.2413
2026-01-16 14:33:30 [DEBUG]
 PromptProcessing: 40.8619
2026-01-16 14:33:35 [DEBUG]
 PromptProcessing: 54.4826
2026-01-16 14:33:41 [DEBUG]
 PromptProcessing: 68.1032
2026-01-16 14:33:46 [DEBUG]
 PromptProcessing: 81.7239
2026-01-16 14:33:52 [DEBUG]
 PromptProcessing: 95.3445
2026-01-16 14:33:55 [DEBUG]
 FinishedProcessingPrompt. Progress: 100

This doesn't happen all the time though. As I progress with the conversation, sometimes it does reuse the cache before generating replies, which start instantly with no delay.

However, even in those situations, if I try to regenerate that reply again it will usually discard the entire cache and reprocess the entire conversation, including the system prompt, from scratch, even though it used the cache just fine with the same prompt to generate the original reply that I'm regenerating.

I didn't try to reproduce this without LM Studio yet. But, from what I read here, it's unlikely I would get a different behavior.

So, my question is, is this particular behavior expected for this type of model? Because the longer the conversation progresses, the more painful it becomes to have the entire cache discarded.

[ggerganov]

The "cache reuse" functionality is different from what you describe.

You are thinking about prefix/prompt caching. This works correctly with llama-server, so if you experience any issue with it, feel free to create a bug report with repro steps.

[pwilkin]

@vnicolici Again: Qwen3Next is a hybrid model. It has recurrent (linear) layers. Recurrent layers don't have a KV cache. They use a single recurrent state which is updated for each token. Therefore, besides the snapshotting which I mentioned above, recurrent states can't do "prefix" cache'ing because they don't keep any sequences :)

@ggerganov Since linear cache takes so much less space than normal KV-cache, it might be reasonable to add a separate snapshot option just for linear layers, because non-linear layers can do normal prefix cache reuse. The logic might be a bit more complicated (since we'd have to check for "existing snapshot + proper prefix"), but it would actually enable something like say an LRU-cache of 2000 linear snapshots done every 100 generated tokens (parametrizable with say --linear-snapshot-cache-size and --linear-snapshot-cache-frequency). Then, for hybrid models, we could use this hybrid snapshot + prefix caching mechanism.

[ggerganov]

@othermod Regarding your earlier comment - I missed the part that you performed your test using LM Studio. If you run the same test with llama-server, you will see that it already works as expected:

$ python test_chunking.py 
Chunk 1... 0.87s
Chunk 2... 1.90s
Chunk 3... 1.92s
Chunk 4... 1.93s
Chunk 5... 1.95s
First instruction... 0.76s
Modified instruction... 0.77s

So my #18497 (comment) that chunking does not work with /v1/chat/completions is not fully correct. Just make sure that your "chunked" requests have max_tokens=0 to avoid generating stuff during the chunking process and it should work as I described earlier in #18497 (comment).

The llama.cpp server has advanced prefix caching techniques that are not yet available in downstream projects.

@pwilkin Yes, this is quite easy to achieve. The logic for deciding when to make a checkpoint is here:

llama.cpp/tools/server/server-context.cpp

Lines 2561 to 2593 in a89002f

	const auto pos_min = llama_memory_seq_pos_min(llama_get_memory(ctx), slot.id);
	const auto pos_max = llama_memory_seq_pos_max(llama_get_memory(ctx), slot.id);
	// no need for empty or small checkpoints
	do_checkpoint = do_checkpoint && (pos_min >= 0 && pos_max >= 64);
	// no need to create checkpoints that are too close together
	do_checkpoint = do_checkpoint && (slot.prompt.checkpoints.empty() || pos_max > slot.prompt.checkpoints.back().pos_max + 64);
	if (do_checkpoint) {
	while (slot.prompt.checkpoints.size() >= (size_t) params_base.n_ctx_checkpoints) {
	// make room for the new checkpoint, if needed
	const auto & cur = slot.prompt.checkpoints.front();
	SLT_WRN(slot, "erasing old context checkpoint (pos_min = %d, pos_max = %d, size = %.3f MiB)\n",
	cur.pos_min, cur.pos_max, (float) cur.data.size() / 1024 / 1024);
	slot.prompt.checkpoints.erase(slot.prompt.checkpoints.begin());
	}
	const size_t checkpoint_size = llama_state_seq_get_size_ext(ctx, slot.id, LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY);
	auto & cur = slot.prompt.checkpoints.emplace_back(server_prompt_checkpoint{
	/*.pos_min = */ pos_min,
	/*.pos_max = */ pos_max,
	/*.data = */ std::vector<uint8_t>(checkpoint_size),
	});
	llama_state_seq_get_data_ext(ctx, cur.data.data(), checkpoint_size, slot.id, LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY);
	SLT_WRN(slot, "created context checkpoint %d of %d (pos_min = %d, pos_max = %d, size = %.3f MiB)\n",
	(int) slot.prompt.checkpoints.size(), params_base.n_ctx_checkpoints, cur.pos_min, cur.pos_max, (float) cur.data.size() / 1024 / 1024);
	}

Currently, it is configured to make a checkpoint at 64 tokens before the full prompt:

llama.cpp/tools/server/server-context.cpp

Lines 2534 to 2539 in a89002f

	// process the last few tokens of the prompt separately in order to allow for a checkpoint to be created.
	if (do_checkpoint && slot.task->n_tokens() - slot.prompt.n_tokens() == 64) {
	break;
	}
	}

[scottix]

llama.cpp/src/llama-memory-hybrid.cpp

Line 179 in 10b26ee

void llama_memory_hybrid::state_write(llama_io_write_i & io, llama_seq_id seq_id, llama_state_seq_flags flags) const {

llama.cpp/src/llama-memory-hybrid-iswa.cpp

Line 185 in 10b26ee

void llama_memory_hybrid_iswa::state_write(llama_io_write_i & io, llama_seq_id seq_id, llama_state_seq_flags flags) const {

I think it is writing and reading partial caches but with this model it needs the full cache.

[sammcj]

Confirming this is an issue with the Qwen 3.5 MoE models as well, not sure if it's helpful but as another data point here's the crash logs:

slot update_slots: id  3 | task 169 | new prompt, n_ctx_slot = 131072, n_keep = -1, task.n_tokens = 29826
slot update_slots: id  3 | task 169 | reusing chunk with size 582, shifting KV cache [22445, 23027) -> [22340, 22922)
/app/src/llama-kv-cache.cpp:403: GGML_ASSERT(hparams.n_pos_per_embd() == 1 && "seq_add() is only supported for n_pos_per_embd() == 1") failed
libggml-base.so.0(+0x1837b)[0x7efd99b1b37b]
libggml-base.so.0(ggml_print_backtrace+0x21f)[0x7efd99b1b7df]
libggml-base.so.0(ggml_abort+0x152)[0x7efd99b1b9b2]
libllama.so.0(+0x1016c7)[0x7efd99cbc6c7]
libllama.so.0(_ZN19llama_memory_hybrid7seq_addEiiii+0x24)[0x7efd99ccfd34]
/app/llama-server(+0x146d9f)[0x562b2cc29d9f]
/app/llama-server(+0x182296)[0x562b2cc65296]
/app/llama-server(+0xa186e)[0x562b2cb8486e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7efd995d0d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7efd995d0e40]
/app/llama-server(+0xa53c5)[0x562b2cb883c5]
2026/02/26 03:11:07 http: proxy error: EOF

Config:

/app/llama-server
  --flash-attn on --threads -1 --threads-batch -1 --threads-http -1
  --prio-batch 2 --prio 2 --slots --metrics
  --cache-type-k q8_0 --cache-type-v q8_0
  --no-context-shift --keep -1 --cache-reuse 256
  --props --no-mmap --cache-ram -1
  --fit on
  --jinja
  --kv-unified
  --n-predict 32768
  --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 48 --draft-max 64
  --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.00 --presence-penalty 0.0 --repeat-penalty 1.0
  --ctx-size 131072
  --model /models/Qwen3.5-35B-A3B-UD-Q6_K_XL.gguf

• Removing --cache-reuse 256 resolves the crash
• llama-server version: 8148 (2446419)