Eval bug: Qwen3.6-35B-A3B MoE — slot hangs in TG after multi-turn cache invalidation; no log output after `create context checkpoint`

[ormandj]

Name and Version

$ /app/llama-server --version
version: 8948 (42401c72b)
built with GNU 13.3.0 for Linux x86_64

Also reproduced on build 8902 (commit 15b243e).

Operating systems

Linux

GGML backends

CUDA

Hardware

• 3x NVIDIA GeForce RTX 3090, compute capability 8.6, 24126 MiB VRAM each
• AMD EPYC 7663
• llama-server runs in a container, no NVLink, no P2P
• GGML_CUDA_NO_PEER_COPY=1, NCCL_P2P_DISABLE=1

Models

• bartowski/Qwen_Qwen3.6-35B-A3B-GGUF Q8_0
• mmproj mmproj-Qwen_Qwen3.6-35B-A3B-f16.gguf

Problem description & steps to reproduce

After multi-turn requests where the prior cache is invalidated ("forcing full prompt re-processing due to lack of cache data"), llama_decode does not return tokens. The slot stays is_processing=true, llamacpp:tokens_predicted_total does not increment, no further log lines appear after create context checkpoint until the client cancels the request.

Only Qwen3.6-35B-A3B (MoE + hybrid attention) hangs. Qwen3.6-27B (dense, also hybrid) hits the same "forcing full prompt re-processing" path repeatedly and decodes normally.

The hang triggers reliably during agent workflows that send multi-turn requests with growing context (e.g. aider's --tries 2 retry sending compiler/test errors back). It does not reproduce when the same final prompt is sent as a single user message without any prior conversation in the same slot.

Server flags:

/app/llama-server
  --model /models/Qwen_Qwen3.6-35B-A3B-Q8_0.gguf
  --mmproj /models/mmproj-Qwen_Qwen3.6-35B-A3B-f16.gguf
  --port 5800
  --ctx-size 262144
  --n-gpu-layers 999
  --parallel 1
  --flash-attn on
  --cache-type-k q8_0
  --cache-type-v q8_0
  -sm layer
  -b 2048
  -ub 2048
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0
  --chat-template-kwargs '{"preserve_thinking":true}'
  --cache-prompt
  --cache-ram -1
  --slot-save-path /models/.cache/lc-qwen3.6-35b-a3b-q8
  --jinja
  --reasoning-format deepseek
  --metrics

Reproduction (observed via aider polyglot benchmark, Rust language slice):

1. Start server with the flags above.
2. Send a code-writing user message + assistant code reply + user follow-up containing compiler/test errors. (See aider 0.86.3 benchmark/benchmark.py driving the polyglot-benchmark Rust exercises with --edit-format whole --threads 1 --tries 2.)
3. After ~3-5 successful exercises, a request hangs.

Hang triggers reliably on these polyglot-benchmark/rust exercises against this model: simple-cipher, alphametics, doubly-linked-list (different exercises depending on prior cache state).

First Bad Commit

Not bisected. Reproduced on 8948 and 8902 (the two builds tested).

Relevant log output

Hang case (task 80862, exact log copy):

slot launch_slot_: id  0 | task 80862 | processing task, is_child = 0
slot update_slots: id  0 | task 80862 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 6190
slot update_slots: id  0 | task 80862 | n_past = 3, slot.prompt.tokens.size() = 5875, seq_id = 0, pos_min = 5874, n_swa = 0
slot update_slots: id  0 | task 80862 | Checking checkpoint with [3009, 3009] against 3...
slot update_slots: id  0 | task 80862 | Checking checkpoint with [965, 965] against 3...
slot update_slots: id  0 | task 80862 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 80862 | erased invalidated context checkpoint (pos_min = 965, pos_max = 965, n_tokens = 966, n_swa = 0, pos_next = 0, size = 62.813 MiB)
slot update_slots: id  0 | task 80862 | erased invalidated context checkpoint (pos_min = 3009, pos_max = 3009, n_tokens = 3010, n_swa = 0, pos_next = 0, size = 62.813 MiB)
slot update_slots: id  0 | task 80862 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 80862 | prompt processing progress, n_tokens = 2048, batch.n_tokens = 2048, progress = 0.330856
slot update_slots: id  0 | task 80862 | n_tokens = 2048, memory_seq_rm [2048, end)
slot update_slots: id  0 | task 80862 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 2048, progress = 0.661712
slot update_slots: id  0 | task 80862 | n_tokens = 4096, memory_seq_rm [4096, end)
slot update_slots: id  0 | task 80862 | prompt processing progress, n_tokens = 4142, batch.n_tokens = 46, progress = 0.669144
slot update_slots: id  0 | task 80862 | n_tokens = 4142, memory_seq_rm [4142, end)
slot update_slots: id  0 | task 80862 | prompt processing progress, n_tokens = 6186, batch.n_tokens = 2044, progress = 0.999354
slot create_check: id  0 | task 80862 | created context checkpoint 1 of 32 (pos_min = 4141, pos_max = 4141, n_tokens = 4142, size = 62.813 MiB)
slot update_slots: id  0 | task 80862 | n_tokens = 6186, memory_seq_rm [6186, end)
slot init_sampler: id  0 | task 80862 | init sampler, took 0.79 ms, tokens: text = 6190, total = 6190
slot update_slots: id  0 | task 80862 | prompt processing done, n_tokens = 6190, batch.n_tokens = 4
slot create_check: id  0 | task 80862 | created context checkpoint 2 of 32 (pos_min = 6185, pos_max = 6185, n_tokens = 6186, size = 62.813 MiB)
[~9 minutes of silence; tokens_predicted_total counter does not increment; slot.is_processing remains true]
[client (litellm) cancels with "net/http: timeout awaiting response headers"]
2026/04/28 00:38:52 http: proxy error: net/http: timeout awaiting response headers
srv          stop: cancel task, id_task = 80862
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot      release: id  0 | task 80862 | stop processing: n_tokens = 113490, truncated = 0

The next log line after created context checkpoint 2 of 32 should be decoding batch, n_tokens = N (debug) followed by the per-token decode pattern. Instead the server is silent. tokens_predicted_total from /metrics confirmed flat at 9544 for the entire interval (sampled every 5 s).

For comparison, the same model + flags decoded successfully on the first request (no prior cache to invalidate, task 7103: 7566 tokens at 120 tok/s) and on the second request after one cache invalidation (task 14684: 1978 tokens at 123 tok/s). The hang appears on a subsequent multi-turn request after additional cache invalidations.

Related issues

• Eval bug: Qwen 3.5 Full prompt re-processing on every conversation turn #20225 — same forcing full prompt re-processing trigger logic. That issue tracks the perf cost of the unnecessary reprocess. This report is the downstream functional hang after the reprocess.
• Server forces full prompt re-processing on subsequent requests (SWA/recurrent memory error) #21831, Eval bug: Server freezes intermittently, /slots endpoint hangs, process ungraceful to stop #20921 — overlapping symptoms across other models (Qwen3.5 122b MoE, gpt-oss 120b, MiniMax M2.5).
• Eval bug: [Bug] Qwen3-Coder-Next (Hybrid) Prompt Cache forces full re-processing due to 'invalidated context checkpoint' despite --swa-full #19794 — --swa-full does not change the behavior on hybrid models. Confirmed here too.

Tested workarounds that did not fix the hang on this build:

• --swa-full
• --reasoning-budget 8192
• --chat-template-kwargs '{"preserve_thinking":true}'
• Bumping request_timeout from 600 to 3600 (just delays the eventual client cancel)

The same prompt body posted directly as a single user message (no prior turns in the slot) does not hang — it generates 16000 reasoning tokens to completion.

[nnyjthm]

Adding evidence — this is reliably reproducible on Qwen3.6-35B-A3B Q8_0 under a specific workload, and I have a WinDbg call stack from a crash variant of the bug.

Setup

• Model: Qwen3.6-35B-A3B-Q8_0.gguf (lmstudio-community quant)
• Binary: official release b8931 (commit 9725a313b)
• Flags: -fa on -ngl 99 -ncmoe 40 --no-mmap --mlock --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified --parallel 1 --jinja -c 32768
• Hardware: RTX 4060 Laptop 8 GB, 64 GB RAM, Windows 11

(I've reproduced the same bug separately at -c 262144 with --parallel 2; reducing context and parallelism does not change the outcome.)

Deterministic repro (4/4 failures)

Pipeline sends three sequential POST /v1/chat/completions calls to the same server. Each prompt is a different chunk of a transcript with no shared prefix with the previous one. First call always succeeds; calls 2 and 3 fail every time.

Minimal shape — call 1 succeeds (long prompt, ~2700 tokens):

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"<chunk 1: ~2700 tokens of transcript A>"}],"max_tokens":2048,"temperature":0.0}'

Call 2 is the same shape but with chunk 2 of the same transcript (no overlap with chunk 1). Server log emits forcing full prompt re-processing due to lack of cache data, then erased invalidated context checkpoint (twice), then created context checkpoint N of 32 — and goes silent or crashes:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"<chunk 2: different ~2700 tokens, no shared prefix>"}],"max_tokens":2048,"temperature":0.0}'

Workarounds tested, none effective:

• --swa-full — no effect
• cache_prompt: false client-side — no effect
• --parallel 1 — no effect (this is our default)
• Increasing timeouts — slot is permanently dead, not slow

Q4 vs Q8 A/B

Run	Model	Result
1–4	Qwen3.6-35B-A3B-Q8_0.gguf	All 4 fail at call 2
5	Qwen3.6-35B-A3B-Q4_K_M.gguf (otherwise identical flags)	All 3 calls succeed

N=1 on Q4 so far — not enough to claim Q4 fixes the bug — but the asymmetry is striking. Most plausibly Q4's faster decode shortens the race window in the slot-cleanup path. Posting in case it helps anyone bisect.

Crash variant — WinDbg call stack

When the bug doesn't hang the slot, it crashes the server with STATUS_STACK_BUFFER_OVERRUN (0xC0000409), but this is the Windows reporting label for __fastfail — the actual exit is via abort(). From the most recent crash dump:

ucrtbase!abort+0x4e
ggml_base!ggml_abort+0xc6
ggml_cuda+0x11f62a              <-- abort callsite
ggml_cuda+0x11d3a9
ggml_cuda!ggml_backend_is_cuda+0x2b964a
ggml_cuda!ggml_backend_is_cuda+0x2b7852
ggml_cuda+0xffde0 -> 0x102e19 -> 0x11eff9 -> 0x121491 -> 0x110ac5
ggml_base!ggml_backend_sched_graph_compute_async+0xafa
llama!llama_set_abort_callback+0x5172 -> +0x4d34 -> +0x67ad   (internals; offsets from nearest exported sym)
llama!llama_decode+0xa
llama_server+0xaeaf1

Recovered from stack memory at rsp+0x70 (the ggml_abort format buffer):

D:\a\llama.cpp\llama.cpp\ggml\src\ggml-cuda\ggml-cuda.cu:97: CUDA error

This is the GGML_ABORT("CUDA error") inside ggml_cuda_error() — meaning a CUDA_CHECK(...) returned non-success. The actual cudaError_t and cudaGetErrorString() text are written to stderr by ggml_cuda_error() before aborting; in my case stderr wasn't captured so I don't have the kernel-level error name. Consistent with an async CUDA error: a prior kernel launched with bad parameters (e.g. out-of-bounds read after position desync between the recurrent state and the attention KV), the driver records the failure asynchronously, and the next CUDA_CHECK call surfaces it.

Suggestion for the next reproducer

If anyone has a chance to repro this with CUDA_LAUNCH_BLOCKING=1 set in the environment and stderr captured, the cudaError_t name from the failing kernel would pin the failure mode (illegal memory access vs. misaligned address vs. invalid resource handle vs. something else). Happy to do this on my end if it'd be useful — let me know.

Build provenance

Path D:\a\llama.cpp\llama.cpp\... in the abort buffer confirms this is the GitHub Actions runner layout — i.e. the official release zip, not a local rebuild.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[haojing8312]

We observed a closely related failure on Qwen3.6-35B-A3B (Unsloth GGUF UD-IQ4_XS + F16 mmproj) on CUDA, 3 x RTX 4090, single slot, context 262144, full GPU offload, flash attention, and the default 8 GiB prompt cache.

This is a production-like multi-turn agent workload, but the evidence below is fully synthetic / sanitized:

• Hermes logged repeated repeated model output detected failures repeatedly from two independent local-Qwen profiles after multi-turn use. The affected services returned garbage repeated characters or stalled; there were no CUDA OOM, Xid, or GPU-reset events.
• Before restarting llama-server, its cache contained five saved prompts, including approximately 89k, 91k, and 68k token contexts (about 7.45 GiB total).
• While the service was in that hot state, a direct /v1/chat/completions request containing only a short synthetic system+user prompt (no business text, no tools, no prior history) returned 64 consecutive / characters in 3/3 runs. The output hash was identical across all three runs.
• After restarting the server and waiting for the model to load, the exact same direct synthetic request completed normally in 3/3 runs (finish_reason=stop, identical normal-output hash).

So this is not a cold-start, single-request reproducer. It is a cache/state-dependent A/B: bad after a long-lived multi-turn cache state, good after process restart. That matches the cache-invalidation direction described in this issue.

Our command is materially:

llama-server -m <Qwen3.6-35B-A3B-UD-IQ4_XS.gguf> --mmproj <mmproj-F16.gguf> \
  -ngl 99 -c 262144 -np 1 -fa on --reasoning off --reasoning-budget 0 \
  --temp 0 --metrics

Unfortunately this local custom CUDA binary reports build_info=b0-unknown, so we cannot yet provide an upstream commit hash. We will first move the canary to a traceable current CUDA build, then repeat the A/B with prompt cache disabled. For the current deployment we added a client-side repeated-output guard and fallback so it fails over instead of leaking the garbage stream.

Does the prompt-cache / recurrent-state issue behind #22450 also explain “hot server emits repeated punctuation for a subsequently unrelated short request, cold restart fixes it”?