Eval bug: Checkpoints and MMProj on Gemma 4 consume abnormal amounts of RAM, leading to llama-server going OOM

[SerialKicked]

Name and Version

version: 8724 (b54cb2e)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows, Linux

GGML backends

CUDA 12, CUDA 13, Vulkan

Hardware

1x RTX3090 24GB
AMD Ryzen 5 3.6Ghz (6 core) + 64GB RAM

Multiple other hardware configurations.

Models

Gemma4
https://huggingface.co/llmfan46/gemma-4-31B-it-uncensored-heretic-GGUF/

Tried many others.

Problem description & steps to reproduce

REVISED ISSUE (see previous edits for history)

With Gemma 4 models (any variant), and Gemma 4 only, checkpoints (ctx-checkpoints) on llama-server are consuming enormous (and increasing) amounts of RAM, to a point that on a machine with 64GB of RAM (and 24GB of VRAM), the entirety of the RAM space can be reached in 4 to 5 full prompt processing of about 16K tokens each. Each generation is making llama-server's RAM consumption bump by several gigabytes.

Workaround: Setting "--ctx-checkpoints" to 1 or 0 with "-np 1" will prevent the server from going out of RAM.

Something notable beyond the OOM issue, is that the memory usage will be different right from the start.

Examples of memory usage with the settings below on a Q4_K_M GGUF

-fa on -kvo -ngl 255 -c 16384 --mmap -m "location/model.gguf" --nowebui

ctx-checkpoints	VRAM (start)	RAM (start)	RAM (1st gen)	RAM (2nd gen)	RAM (after 3rd gen)
(defaults)	23.6 / 24 GB	0.7GB (+ file)	10GB (+ file)	18GB (+ file)	OOM (>25GB + File)
set to 1 + np 1	21.6 / 24 GB	0.7GB (+ file)	1.5GB (+ file)	1.5GB (+ file)	1.5GB (+ file)
set to 0 + np 1	21.6 / 24 GB	0.4GB (+ file)	0.4GB (+ file)	0.4GB (+ file)	0.4GB (+ file)

(each generation is a different 16K tokens long chatlog sent for text or chat completion processing to the server)

Notes:

• The "-np" setting seems to have a multiplicative value to the consumed VRAM. The higher, the more RAM is used.
• The "-cram" flag doesn't seem to have any impact either way (set or unset, same values)
• The server behaves normally with other architectures (test Qwen 3.5 and Mistral)

First Bad Commit

At least since b8660. Bug is consistent since the initial implementation of Gemma 4.

Relevant log output

Not super relevant anymore, but left as is, just in case.

Logs

One log (crashed during processing i think)

srv  params_from_: Grammar lazy: false
srv  params_from_: Generation prompt: ''
res  add_waiting_: add task 891 to waiting list. current waiting = 0 (before add)
que          post: new task, id = 891/1, front = 0
que    start_loop: processing new tasks
que    start_loop: processing task, id = 891
slot get_availabl: id  2 | task -1 | selected slot by LRU, t_last = -1
srv  get_availabl: updating prompt cache
srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 20480 tokens, 8589934592 est)
srv  get_availabl: prompt cache update took 0.01 ms
slot launch_slot_: id  2 | task -1 | launching slot : {"id":2,"n_ctx":20480,"speculative":false,"is_processing":false}
slot launch_slot_: id  2 | task -1 | sampler chain: logits -> ?penalties -> dry -> ?top-n-sigma -> ?top-k -> ?typical -> ?top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  2 | task 891 | processing task, is_child = 0
slot slot_save_an: id  3 | task -1 | saving idle slot to prompt cache
slot slot_save_an: id  3 | task -1 | __TEST_TAG_CLEAR_IDLE_SLOT__
srv   prompt_save:  - saving prompt with length 18674, total state size = 5059.174 MiB
slot prompt_clear: id  3 | task -1 | clearing prompt with 18674 tokens
srv        update:  - cache state: 1 prompts, 19459.390 MiB (limits: 8192.000 MiB, 20480 tokens, 20480 est)
srv        update:    - prompt 000001B16FFD4730:   18674 tokens, checkpoints:  4, 19459.390 MiB
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 892, front = 0
slot update_slots: id  2 | task 891 | new prompt, n_ctx_slot = 20480, n_keep = 0, task.n_tokens = 17702
slot update_slots: id  2 | task 891 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  2 | task 891 | prompt processing progress, n_tokens = 2048, batch.n_tokens = 2048, progress = 0.115693
srv  update_slots: decoding batch, n_tokens = 2048
set_adapters_lora: adapters = 0000000000000000
adapters_lora_are_same: adapters = 0000000000000000
set_embeddings: value = 0
ggml_backend_cuda_graph_compute: CUDA graph warmup reset
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 892
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 893, front = 0
slot update_slots: id  2 | task 891 | n_tokens = 2048, memory_seq_rm [2048, end)
slot update_slots: id  2 | task 891 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 2048, progress = 0.231386
srv  update_slots: decoding batch, n_tokens = 2048
set_adapters_lora: adapters = 0000000000000000
adapters_lora_are_same: adapters = 0000000000000000
set_embeddings: value = 0
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 893
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 894, front = 0
slot update_slots: id  2 | task 891 | n_tokens = 4096, memory_seq_rm [4096, end)
slot update_slots: id  2 | task 891 | prompt processing progress, n_tokens = 6144, batch.n_tokens = 2048, progress = 0.347079
srv  update_slots: decoding batch, n_tokens = 2048
set_adapters_lora: adapters = 0000000000000000
adapters_lora_are_same: adapters = 0000000000000000
set_embeddings: value = 0
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 894
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 895, front = 0
slot update_slots: id  2 | task 891 | n_tokens = 6144, memory_seq_rm [6144, end)
slot update_slots: id  2 | task 891 | prompt processing progress, n_tokens = 8192, batch.n_tokens = 2048, progress = 0.462773
srv  update_slots: decoding batch, n_tokens = 2048
set_adapters_lora: adapters = 0000000000000000
adapters_lora_are_same: adapters = 0000000000000000
set_embeddings: value = 0
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 895
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 896, front = 0
slot update_slots: id  2 | task 891 | n_tokens = 8192, memory_seq_rm [8192, end)
slot update_slots: id  2 | task 891 | 8192 tokens since last checkpoint at 0, creating new checkpoint during processing at position 10240
slot update_slots: id  2 | task 891 | prompt processing progress, n_tokens = 10240, batch.n_tokens = 2048, progress = 0.578466
slot update_slots: id  2 | task 891 | created context checkpoint 1 of 32 (pos_min = 3584, pos_max = 8191, n_tokens = 8192, size = 3600.054 MiB)
srv  update_slots: decoding batch, n_tokens = 2048
set_adapters_lora: adapters = 0000000000000000
adapters_lora_are_same: adapters = 0000000000000000
set_embeddings: value = 0
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 896
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 897, front = 0
slot update_slots: id  2 | task 891 | n_tokens = 10240, memory_seq_rm [10240, end)
slot update_slots: id  2 | task 891 | prompt processing progress, n_tokens = 12288, batch.n_tokens = 2048, progress = 0.694159
srv  update_slots: decoding batch, n_tokens = 2048
set_adapters_lora: adapters = 0000000000000000
adapters_lora_are_same: adapters = 0000000000000000
set_embeddings: value = 0
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 897
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 898, front = 0
slot update_slots: id  2 | task 891 | n_tokens = 12288, memory_seq_rm [12288, end)
slot update_slots: id  2 | task 891 | prompt processing progress, n_tokens = 14336, batch.n_tokens = 2048, progress = 0.809852
srv  update_slots: decoding batch, n_tokens = 2048
set_adapters_lora: adapters = 0000000000000000
adapters_lora_are_same: adapters = 0000000000000000
set_embeddings: value = 0
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 898
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 899, front = 0
slot update_slots: id  2 | task 891 | n_tokens = 14336, memory_seq_rm [14336, end)
slot update_slots: id  2 | task 891 | prompt processing progress, n_tokens = 16384, batch.n_tokens = 2048, progress = 0.925545
srv  update_slots: decoding batch, n_tokens = 2048
set_adapters_lora: adapters = 0000000000000000
adapters_lora_are_same: adapters = 0000000000000000
set_embeddings: value = 0
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 899
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 900, front = 0
slot update_slots: id  2 | task 891 | n_tokens = 16384, memory_seq_rm [16384, end)
slot update_slots: id  2 | task 891 | 8192 tokens since last checkpoint at 8192, creating new checkpoint during processing at position 17186
slot update_slots: id  2 | task 891 | prompt processing progress, n_tokens = 17186, batch.n_tokens = 802, progress = 0.970851
slot update_slots: id  2 | task 891 | created context checkpoint 2 of 32 (pos_min = 11776, pos_max = 16383, n_tokens = 16384, size = 3600.054 MiB)
srv  update_slots: decoding batch, n_tokens = 802
set_adapters_lora: adapters = 0000000000000000
adapters_lora_are_same: adapters = 0000000000000000
set_embeddings: value = 0
srv  update_slots: run slots completed
que    start_loop: waiting for new tasks
que    start_loop: processing new tasks
que    start_loop: processing task, id = 900
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 901, front = 0
slot update_slots: id  2 | task 891 | n_tokens = 17186, memory_seq_rm [17186, end)
slot update_slots: id  2 | task 891 | prompt processing progress, n_tokens = 17698, batch.n_tokens = 512, progress = 0.999774

This one crashes right at the start.

srv  params_from_: Grammar lazy: false
srv  params_from_: Generation prompt: ''
res  add_waiting_: add task 3475 to waiting list. current waiting = 0 (before add)
que          post: new task, id = 3475/1, front = 0
que    start_loop: processing new tasks
que    start_loop: processing task, id = 3475
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
srv  get_availabl: updating prompt cache
srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
srv        update:  - cache state: 1 prompts, 12377.916 MiB (limits: 8192.000 MiB, 24576 tokens, 24576 est)
srv        update:    - prompt 000001FCBE9A81D0:   22401 tokens, checkpoints:  5, 12377.916 MiB
srv  get_availabl: prompt cache update took 0.01 ms
slot launch_slot_: id  0 | task -1 | launching slot : {"id":0,"n_ctx":24576,"speculative":false,"is_processing":false}
slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> dry -> ?top-n-sigma -> ?top-k -> ?typical -> ?top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  0 | task 3475 | processing task, is_child = 0
slot slot_save_an: id  1 | task -1 | saving idle slot to prompt cache
slot slot_save_an: id  1 | task -1 | __TEST_TAG_CLEAR_IDLE_SLOT__
srv   prompt_save:  - saving prompt with length 22901, total state size = 2863.297 MiB

[blakkd]

I also observe this memory leak, RAM usage grows extremely fast, RTX 3090 (Ampere) happens at least on CUDA and Vulkan, I tried both

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.20             Driver Version: 580.126.20     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        On  |   00000000:06:00.0  On |                  N/A |
|  0%   41C    P8             20W /  380W |   23931MiB /  24576MiB |      4%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

/home/user/llama.cpp/build/bin/llama-server \
      --model /mnt/storage/GGUFs/gemma-4/gemma-4-31B-it-UD-Q3_K_XL.gguf \
      # --mmproj /mnt/storage/GGUFs/gemma-4/mmproj-BF16.gguf \
      --webui-mcp-proxy \
      --no-warmup \
      --ctx-size 52000 \
      --no-context-shift \
      --n-gpu-layers 61 \
      --temp 1 \
      --top-p 0.95 \
      --top-k 64 \
      --min-p 0 \
      --presence-penalty 0 \
      --repeat-penalty 1 \
      --jinja \
      --host 0.0.0.0 \
      --port ${PORT} \
      --flash-attn on \
      --mlock \
      # --backend-sampling \
      # --chat-template-kwargs '{"enable_thinking":false}'

[aldehir]

Try reducing the number of checkpoints.

[SerialKicked]

Try reducing the number of checkpoints.

That's the first thing I tried, I assumed it was tied to the checkpoints, but it's not, at least not directly.
This has no effect. 3, 8, or 32 checkpoints don't change a thing. The -cram option is also ignored.

Also, when we say "it grows fast", RAM usage balloons by several GB per new prompt being processed.

[ColbyB722]

Does this happen with version b8660 for you?

[SerialKicked]

Does this happen with version b8660 for you?

Yes, at least for me (seems even worse at it)

[Offset0x]

--cache-ram 0 --ctx-checkpoints 1 fixes the RAM issue for me.
For VRAM, you should use -np 1 and honestly, -ctk q4_0 -ctv q4_0 works pretty great too.

[SerialKicked]

--cache-ram 0 --ctx-checkpoints 1 fixes the RAM issue for me.

This seems to negate the leak, yes. So I guess we can assume that cram is either all or nothing (max value is ignored) or/and that checkpoints are leaking memory like crazy on Gemma 4.

Double checked against Qwen 3.5 and Magistral 24B, and neither have the issue.

[Atliac]

I have conducted some tests regarding this issue. It appears that --cache-ram can be used as desired without issue. However, I noticed that --ctx-checkpoints (defaulting to 32) leads to very high memory consumption. By setting --ctx-checkpoints to 1 (or 0), the memory usage stabilizes, and we can still benefit from the KV cache as long as --cache-ram is enabled. It seems the checkpointing mechanism in Gemma 4 requires much more memory than expected.

[blakkd]

By setting --ctx-checkpoints to 1 (or 0)

confirmed working, letting --cache-ram at the 8192 default

[hql1229]

Same problem solved By setting --ctx-checkpoints to 1 (or 0).

[SerialKicked]

Okay, I'll edit the op to reflect the proper issue (and workaround) as soon as I'm back in front of a computer and not a phone tomorrow, for clarity sake.

[andysalerno]

I'm curious if this is a bug, or an unfortunate fact of life related to Gemma's implementation. Hoping it's just a bug that can be fixed!