Eval bug: Crash: ggml_new_object: not enough space in context's memory pool with -sm tensor after long context

[jmig1109]

Name and Version

llama-server --version
version: 9202 (e0de4c2)
built with GNU 13.3.0 for Linux x86_64

Build command:

cmake -B build   -DGGML_NATIVE=ON   -DGGML_CUDA=ON   -DCMAKE_CUDA_COMPILER="/usr/local/cuda-13.1/bin/nvcc"   -DCMAKE_PREFIX_PATH="/usr/local/cuda-13.1"   -DCMAKE_INSTALL_RPATH="/usr/local/cuda-13.1/lib64;\$ORIGIN"   -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON -DGGML_CUDA_FORCE_CUBLAS=true -DGGML_SCHED_MAX_COPIES=8
cmake --build build --config Release -j 8

LLama-server launch command:

CUDA_VISIBLE_DEVICES=1,0,2 CUDA_SCALE_LAUNCH_QUEUES=4x llama-server --models-preset /mnt/fast1/gguf-models/models-build.ini --host 0.0.0.0 --port 8080 --models-max 1 -fa on --log-colors on -cram 32768 --slot-save-path /mnt/fast1/kv-cache-storage --cache-prompt --jinja --cache-idle-slots --log-prefix --log-timestamps --log-file /docker/llama.cpp/logs --perf --keep -1 --context-shift --cache-reuse 1 --metrics

Operating systems

Linux

GGML backends

CUDA

Hardware

CPU: AMD Ryzen 7 5700X (16) @ 4.665GHz
GPU: NVIDIA GeForce RTX 3090
GPU: NVIDIA GeForce RTX 3090
GPU: NVIDIA RTX PRO 4000 Blackwell
Memory: 128714MiB
Nvidia Driver: 595.71.05
OS: Pop!_OS 24.04 LTS
Kernel: Linux 6.18.7-76061807-generic (x86_64)

Models

am17an/Qwen3.6-27B-MTP-GGUF - the same version used to test the MTP pull request.
This is a Q8_0 quantization.

model presets:

[Qwen3.6-27B-MTP-Q8]
model = /mnt/fast1/gguf-models/Qwen3.6-27B/Qwen3.6-27B-MTP-Q8_0.gguf
mmproj = /mnt/fast1/gguf-models/Qwen3.6-27B/mmproj-F16.gguf
ngl = 99
ctx-size = 0
np = 4
kvo = True
kvu = True
temp = 0.6
top-p = 0.95
presence-penalty = 0.0
repeat-penalty = 1.0
top-k = 20
min-p = 0.0
sleep-idle-seconds = 360
no-prefill-assistant = true
chat-template-kwargs = {"preserve_thinking": true}
sm = tensor
spec-type = draft-mtp
spec-draft-n-max = 2
b = 8192
ub = 1024

Problem description & steps to reproduce

##Description:
The llama-server crashes after long conversations with the Qwen3.6-27B-MTP-Q8 model using --spec-type draft-mtp. The crash always occurs during prompt processing when a new context checkpoint is being created.

cache size limit reached, removing oldest entry (size = 22200.419 MiB)
created context checkpoint 1 of 32 (pos_min = 8191, pos_max = 8191, n_tokens = 8192, size = 181.782 MiB)
GGML_ASSERT(obj_new) failed
ggml_new_object: not enough space in the context's memory pool (needed 1073741936, available 1073741824)

##Reproduction:
Start llama-server with Qwen3.6-27B-MTP-Q8 and --spec-type draft-mtp
Run multiple long chats exhausitng kv cache
Server crashes during prompt processing
Run multiple long chats exhausitng kv cache

##Logs:

at startup:
common_init_from_params: KV cache shifting is not supported for this context, disabling KV cache shifting
I imagine this must play a role (together with -kvu and -np 4).

First Bad Commit

No response

Relevant log output

Logs

[logs-llama-cpp.txt](https://github.com/user-attachments/files/27954337/logs-llama-cpp.txt)

[biturbo9]

Additional isolation data — 2× RTX 5090, Windows 11, CUDA 13.1, llama.cpp b9209 (Windows-CUDA build)

Confirming this on Qwen3.6-27B MTP and adding a reproduction plus isolation that may help narrow it down.

Repro

Long multi-turn soak: repeated ~70K-token prompts across fresh conversations (divergent prefixes), with

--spec-type draft-mtp --spec-draft-n-max 4 --parallel 1 --no-context-shift --cache-ram 2048

Crashes reliably after ~6–7 cumulative large runs.

Crash

ggml.c:1766: GGML_ASSERT(obj_new) failed
ggml_new_object: not enough space in the context's memory pool (needed 1073741936, available 1073741824)

The fixed 1 GiB ggml object pool, overrun by exactly 112 bytes — and the overrun value is identical regardless of context size.

What isolates it

Variant (same model, same soak)	Result
non-MTP (no --spec-type draft-mtp), --split-mode layer	12/12 runs clean (60 requests)
MTP, --split-mode tensor	crash above, ~run 6–7
MTP, --split-mode tensor, ctx 131072 vs 262144	identical crash, same run, same 112-byte overrun (context-independent)
MTP, --split-mode layer	silent process exit (no GGML_ASSERT) ~run 3; log ends mid prompt_save: total state size = ~4.87 GB
MTP, --cache-ram 0	different crash at warmup: ggml-backend.cpp:194: GGML_ASSERT(buffer) (never loads)

So it appears specific to the draft-mtp speculative path — not the model itself, not context size, and not tunable via --cache-ram (0 = warmup assert, 2048 = soak assert).

Possibly relevant

The RAM prompt cache grows far past its configured limit with MTP:

cache state: 1 prompts, 8504.588 MiB (limits: 2048.000 MiB ...)
saving prompt with length 71038, total state size = 4869.704 MiB (draft: 278.847 MiB)

Context checkpoints reach ~427 MiB each, and every divergent conversation forces full re-processing (hybrid/recurrent, #13194), which rebuilds the full checkpoint set each time. This looks like an interaction between the prompt-cache rewrite (#16391) and the draft-mtp + qwen35 hybrid context-checkpoint path. Related: #22404.

Happy to provide full logs / per-run traces.

[Atomic-Germ]

Wait does drafting actually work in any case at all!?

[annagorshunova]

It seems to be a ggml issue rather than a llama.cpp issue.

When using sm = tensor (--split-mode tensor), buffer_reset() resets the nested backend buffers but does not reset the meta simple_ctx. As a result, simple tensor objects allocated in simple_ctx remain allocated across graph rebuilds, so ggml_used_mem(simple_ctx) keeps growing until the context memory pool is exhausted.

This patch for 49c21f97 was prepared with the help of Codex and Qwen 3.6 27B itself. It resets the meta simple contexts and clears the related meta tensor caches.

ggml-backend-meta-fix.patch

[jsmcconn]

I've also been seeing exactly this issue, similarly i asked qwen3.6 27b to fix it and this is the change it suggested: cca2b95

[slavap]

I have the same issue, it occurs not only with MTP, but every time -sm tensor used.

[jmig1109]

Thanks @annagorshunova and @jsmcconn for investigating the issue further! I have edited the title to highlight that it was most likely a -sm tensor issue related to ggml , not specifically MTP issue.

I'm however unsure how to proceed from here - if the issue should be filed under ggml and closed here, or if someone specific should be tagged. Would appreciate some guidance.

[annagorshunova]

@jmig1109 Probably it makes sense to keep this issue here for now, because it is triggered through llama.cpp and the relevant option (--split-mode tensor) is also in llama.cpp.

Also, I am not completely sure about my patch. It works for me, but maybe the reset should clear the meta tensor state unconditionally (simple_tensors and split_state_cache), as @jsmcconn proposed. That is why I posted a patch rather than opening a PR.

This needs a review from someone familiar with the ggml meta backend and buffer reset details. Maybe @JohannesGaessler could take a look, based on the CODEOWNERS entry for this file?

[JohannesGaessler]

I was busy with things unrelated to llama.cpp/ggml for the last 1-2 weeks but the issue here should be fixed by this PR: #22616

[jsmcconn]

Thanks @JohannesGaessler, do you expect to address this fixme in that PR? I found it necessary to reset the split state cache to avoid oom on repeated long context queries. Although admittedly I don't fully understand the architecture.

+    // FIXME
+    // The size of the split state cache is unbounded and can theoretically grow infinitely large.
+    // However, it is also expensive to build and clearing it on every rebuild in ggml_backend_meta_graph_compute is too expensive.

https://github.com/JohannesGaessler/llama.cpp/blob/6990589e8ca3579bcc07b989bf839da9b3f87ec8/ggml/src/ggml-backend-meta.cpp#L425-L429

[qits-kkruse]

Wait does drafting actually work in any case at all!?

It's a thing now, git pull and recompile.