Eval bug: forcing full prompt re-processing due to lack of cache data unsloth/Qwen3.5-2B-MTP-GGUF:Q4_K_XL

[thomas-0816]

Name and Version

./llama-server --version
version: 9642 (3686e9d)
built with GNU 11.4.0 for Linux x86_64

./llama-server -hf unsloth/Qwen3.5-2B-MTP-GGUF:Q4_K_XL --no-mmproj --reasoning off

full prompt re-processing with every request, slow for bigger prompts

2.00.280.993 I srv  params_from_: Chat format: peg-native
2.00.281.274 I slot get_availabl: id  3 | task -1 | selected slot by LCP similarity, sim_best = 0.500 (> 0.100 thold), f_keep = 0.056
2.00.281.276 I srv  get_availabl: updating prompt cache
2.00.281.784 W srv   prompt_save:  - saving prompt with length 178, total state size = 21.356 MiB (draft: 0.000 MiB)
2.00.310.595 I srv          load:  - looking for better prompt, base f_keep = 0.056, sim = 0.500
2.00.310.604 I srv        update:  - cache state: 7 prompts, 285.314 MiB (limits: 8192.000 MiB, 262144 tokens, 262144 est)
2.00.310.605 I srv        update:    - prompt 0x619d01a2e150:     478 tokens, checkpoints:  1,    44.143 MiB
2.00.310.606 I srv        update:    - prompt 0x619d090dfe60:      51 tokens, checkpoints:  1,    39.131 MiB
2.00.310.606 I srv        update:    - prompt 0x619d01a2a110:     119 tokens, checkpoints:  1,    39.929 MiB
2.00.310.606 I srv        update:    - prompt 0x619d01a29900:     192 tokens, checkpoints:  1,    40.786 MiB
2.00.310.607 I srv        update:    - prompt 0x619d01a28e30:     104 tokens, checkpoints:  1,    39.753 MiB
2.00.310.607 I srv        update:    - prompt 0x619d01a27380:     206 tokens, checkpoints:  1,    40.950 MiB
2.00.310.607 I srv        update:    - prompt 0x619d0708d030:     178 tokens, checkpoints:  1,    40.622 MiB
2.00.310.609 I srv  get_availabl: prompt cache update took 29.33 ms
2.00.310.681 I reasoning-budget: activated, budget=2147483647 tokens
2.00.310.682 I reasoning-budget: deactivated (natural end)
2.00.310.715 I slot launch_slot_: id  3 | task 1207 | processing task, is_child = 0
2.00.310.717 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
2.00.310.718 I slot prompt_clear: id  0 | task -1 | clearing prompt with 0 tokens
2.00.310.842 I slot process_sing: id  1 | task -1 | saving idle slot to prompt cache
2.00.310.844 I slot prompt_clear: id  1 | task -1 | clearing prompt with 0 tokens
2.00.310.959 I slot process_sing: id  2 | task -1 | saving idle slot to prompt cache
2.00.310.960 I slot prompt_clear: id  2 | task -1 | clearing prompt with 0 tokens
2.00.311.085 I slot update_slots: id  3 | task 1207 | Checking checkpoint with [16, 16] against 10...
2.00.311.087 W slot update_slots: id  3 | task 1207 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
2.00.311.088 W slot update_slots: id  3 | task 1207 | erased invalidated context checkpoint (pos_min = 16, pos_max = 16, n_tokens = 17, n_swa = 0, pos_next = 0, size = 19.266 MiB)
2.00.403.309 I slot create_check: id  3 | task 1207 | created context checkpoint 1 of 32 (pos_min = 15, pos_max = 15, n_tokens = 16, size = 19.266 MiB)
2.02.534.659 I slot print_timing: id  3 | task 1207 | n_decoded =    100, tg =  47.51 t/s
2.04.199.532 I slot print_timing: id  3 | task 1207 | prompt eval time =     118.88 ms /    20 tokens (    5.94 ms per token,   168.24 tokens per second)
2.04.199.536 I slot print_timing: id  3 | task 1207 |        eval time =    3769.54 ms /   177 tokens (   21.30 ms per token,    46.96 tokens per second)
2.04.199.536 I slot print_timing: id  3 | task 1207 |       total time =    3888.42 ms /   197 tokens
2.04.199.537 I slot print_timing: id  3 | task 1207 |    graphs reused =       1357
2.04.199.559 I slot      release: id  3 | task 1207 | stop processing: n_tokens = 196, truncated = 0
2.04.199.577 I srv  update_slots: all slots are idle


2.10.519.351 I srv  params_from_: Chat format: peg-native
2.10.519.512 I slot get_availabl: id  3 | task -1 | selected slot by LCP similarity, sim_best = 0.524 (> 0.100 thold), f_keep = 0.056
2.10.519.514 I srv  get_availabl: updating prompt cache
2.10.519.705 W srv   prompt_save:  - saving prompt with length 196, total state size = 21.567 MiB (draft: 0.000 MiB)
2.10.547.211 I srv          load:  - looking for better prompt, base f_keep = 0.056, sim = 0.524
2.10.547.222 I srv        update:  - cache state: 8 prompts, 326.147 MiB (limits: 8192.000 MiB, 262144 tokens, 262144 est)
2.10.547.222 I srv        update:    - prompt 0x619d01a2e150:     478 tokens, checkpoints:  1,    44.143 MiB
2.10.547.223 I srv        update:    - prompt 0x619d090dfe60:      51 tokens, checkpoints:  1,    39.131 MiB
2.10.547.223 I srv        update:    - prompt 0x619d01a2a110:     119 tokens, checkpoints:  1,    39.929 MiB
2.10.547.224 I srv        update:    - prompt 0x619d01a29900:     192 tokens, checkpoints:  1,    40.786 MiB
2.10.547.224 I srv        update:    - prompt 0x619d01a28e30:     104 tokens, checkpoints:  1,    39.753 MiB
2.10.547.224 I srv        update:    - prompt 0x619d01a27380:     206 tokens, checkpoints:  1,    40.950 MiB
2.10.547.225 I srv        update:    - prompt 0x619d0708d030:     178 tokens, checkpoints:  1,    40.622 MiB
2.10.547.225 I srv        update:    - prompt 0x619d0916baf0:     196 tokens, checkpoints:  1,    40.833 MiB
2.10.547.226 I srv  get_availabl: prompt cache update took 27.71 ms
2.10.547.302 I reasoning-budget: activated, budget=2147483647 tokens
2.10.547.302 I reasoning-budget: deactivated (natural end)
2.10.547.338 I slot launch_slot_: id  3 | task 1386 | processing task, is_child = 0
2.10.547.338 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
2.10.547.340 I slot prompt_clear: id  0 | task -1 | clearing prompt with 0 tokens
2.10.547.499 I slot process_sing: id  1 | task -1 | saving idle slot to prompt cache
2.10.547.500 I slot prompt_clear: id  1 | task -1 | clearing prompt with 0 tokens
2.10.547.611 I slot process_sing: id  2 | task -1 | saving idle slot to prompt cache
2.10.547.613 I slot prompt_clear: id  2 | task -1 | clearing prompt with 0 tokens
2.10.547.736 I slot update_slots: id  3 | task 1386 | Checking checkpoint with [15, 15] against 11...
2.10.547.737 W slot update_slots: id  3 | task 1386 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
2.10.547.738 W slot update_slots: id  3 | task 1386 | erased invalidated context checkpoint (pos_min = 15, pos_max = 15, n_tokens = 16, n_swa = 0, pos_next = 0, size = 19.266 MiB)
2.10.640.952 I slot create_check: id  3 | task 1386 | created context checkpoint 1 of 32 (pos_min = 16, pos_max = 16, n_tokens = 17, size = 19.266 MiB)
2.11.371.039 I slot print_timing: id  3 | task 1386 | prompt eval time =     119.08 ms /    21 tokens (    5.67 ms per token,   176.35 tokens per second)
2.11.371.043 I slot print_timing: id  3 | task 1386 |        eval time =     704.20 ms /    33 tokens (   21.34 ms per token,    46.86 tokens per second)
2.11.371.044 I slot print_timing: id  3 | task 1386 |       total time =     823.28 ms /    54 tokens
2.11.371.045 I slot print_timing: id  3 | task 1386 |    graphs reused =       1388
2.11.371.066 I slot      release: id  3 | task 1386 | stop processing: n_tokens = 53, truncated = 0
2.11.371.083 I srv  update_slots: all slots are idle

ref #13194

Operating systems

Linux

GGML backends

Vulkan

Hardware

Ryzen 7840U

Models

unsloth/Qwen3.5-2B-MTP-GGUF:Q4_K_XL

Problem description & steps to reproduce

send a prompt

send the same prompt with 1 extra letter

First Bad Commit

No response

Relevant log output

Logs

2.10.547.737 W slot update_slots: id  3 | task 1386 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
2.10.547.738 W slot update_slots: id  3 | task 1386 | erased invalidated context checkpoint (pos_min = 15, pos_max = 15, n_tokens = 16, n_swa = 0, pos_next = 0, size = 19.266 MiB)

[thomas-0816]

same issue with unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL

no issue with unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL, unsloth/gemma-4-E4B-it-qat-GGUF:UD-Q4_K_XL

[jacekpoplawski]

You are sending 20 tokens if I am correct. Could you clarify the usecase? Is this the first prompt and the second prompt in the chat?

[thomas-0816]

here is the full case:

./llama-server -hf unsloth/Qwen3.5-2B-GGUF:Q4_K_XL --no-mmproj --reasoning off

curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? A"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? AB"}]}' | jq .

Logs containing "forcing full prompt re-processing due to lack of cache data":

logs.txt

[jacekpoplawski]

But this is the first prompt and you are changing it, there is nothing to cache.
What is the expected behaviour (what happens when you observe "no issue")?

[thomas-0816]

The "forcing full prompt re-processing" error comes on the second and third prompt, all 3 have a common prefix.

When you run the same test with 3 prompts having a common prefix with Gemma4 E4B, you see in the logs, that there is no "full prompt re-processing" and prompt_save / restored context checkpoint work correctly.

./llama-server -hf unsloth/gemma-4-E4B-it-qat-GGUF:UD-Q4_K_XL --no-mmproj --reasoning off

curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? A"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? AB"}]}' | jq .

Logs NOT containing "forcing full prompt re-processing due to lack of cache data" with Gemma4:

logs2.txt

For bigger prompts, the "full prompt re-processing" error hurts the performance badly.

[thomas-0816]

cc @pwilkin

[thomas-0816]

here is a screenshot of the logs side by side using qwen3.5 on the left and gemma4 on the right running the same 3 prompts:

[jacekpoplawski]

The difference is “against 9” vs “against 0”. “9” leads to invalidation, while “0” is empty.

Actually, the right side looks strange, because a checkpoint should not be created at 0.

[thomas-0816]

see #22746

[AbelVM]

Same issue with Qwen3.6-35b-a3b-mtp and llama.cpp b9694

[thomas-0816]

More testing: issue happens with Vulkan and Cpu-only

When sending messages with same prefix, "forcing full prompt re-processing" happens:

• ["When was Beethoven born?"]
• ["When was Beethoven born? A"]
• ["When was Beethoven born? AB"]

When sending multiple messages, "forcing full prompt re-processing" does not happen

• ["When was Beethoven born?"]
• ["When was Beethoven born?", "A"]
• ["When was Beethoven born?", "A", "B"]

To reproduce:
/llama-server -hf unsloth/Qwen3.5-2B-GGUF:Q4_K_XL --no-mmproj --reasoning off

Problem happens:
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? A"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? AB"}]}' | jq .

Problem does not happen:
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}, {"role": "user", "content": "ABC"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}, {"role": "user", "content": "ABC"}, {"role": "user", "content": "DEF"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}, {"role": "user", "content": "ABC"}, {"role": "user", "content": "DEF"}, {"role": "user", "content": "GHI"}]}' | jq .

[jacekpoplawski]

As I wrote this is expected behavior, there is no bug.

[thomas-0816]

And why does it happen with Qwen3.5/3.6 and not with Gemma4?

[jacekpoplawski]

Something is wrong with the Gemma log, because checkpoint shouldn't be created at zero. But I would need to reproduce/debug it to tell you more