Eval bug: GLM-4.7-Flash flash attention error on (long?) prompts with KV quantization

[SharkWipf]

Name and Version

(0) sebastiaan@almavm:~/src/llama.cpp $ ./build/bin/llama-server --version
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
version: 7813 (51fa458a9)
built with GNU 14.3.1 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

Ryzen 5950X (not used, fully offloaded) + RTX 3090

Models

Unsloth GLM-4.7-Flash-UD-Q4_K_XL.gguf (Post Jan 21 update)

Problem description & steps to reproduce

Build command:
cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_CUDA_ENABLE_UNIFIED_MEMORY=1 && cmake --build build --config Release -j

Launch command (minimal reproduction):
./build/bin/llama-server -m models/GLM-4.7-Flash-UD-Q4_K_XL.gguf --n-gpu-layers 1100 -c $((202752/2)) --flash-attn on --cache-type-k q4_1 --cache-type-v q4_1
Also tried with q4_0.
(Edit: No problems without kv quant btw)
(Edit 2: Issue also happens with q8_0)

Seems to work fine for short/simple prompts, but once I throw in a ~79k token prompt (or possibly something specific in that prompt), it goes through the whole KV cache processing fine for the most part, but then hits a crash by the end.
The prompt contents is mostly a map of a large codebase and some instructions.

First Bad Commit

No response

Relevant log output

Logs

main: model loaded                                                                                                                     
main: server is listening on http://127.0.0.1:8080                                                                                     
main: starting the main loop...                                                                                                        
srv  update_slots: all slots are idle                                                                                                  
srv  log_server_r: done request: GET / 127.0.0.1 200                                                                                   
srv  params_from_: Chat format: GLM 4.5                                                                                                
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1                                                      
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0                                                                      
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 101376, n_keep = 0, task.n_tokens = 78995
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)                                                    
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 2048, batch.n_tokens = 2048, progress = 0.025926
slot update_slots: id  3 | task 0 | n_tokens = 2048, memory_seq_rm [2048, end)                                              
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 2048, progress = 0.051851

[...]

slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 57344, batch.n_tokens = 2048, progress = 0.725919                                                                                                                                01:47:57 [14/1840]
slot update_slots: id  3 | task 0 | n_tokens = 57344, memory_seq_rm [57344, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 59392, batch.n_tokens = 2048, progress = 0.751845
slot update_slots: id  3 | task 0 | n_tokens = 59392, memory_seq_rm [59392, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 61440, batch.n_tokens = 2048, progress = 0.777771
slot update_slots: id  3 | task 0 | n_tokens = 61440, memory_seq_rm [61440, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 63488, batch.n_tokens = 2048, progress = 0.803696
slot update_slots: id  3 | task 0 | n_tokens = 63488, memory_seq_rm [63488, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 65536, batch.n_tokens = 2048, progress = 0.829622
/home/sebastiaan/src/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:96: CUDA error
CUDA error: invalid argument
  current device: 0, in function launch_fattn at /home/sebastiaan/src/llama.cpp/ggml/src/ggml-cuda/template-instances/../fattn-common.cuh:1004
  cudaGetLastError()
[New LWP 14401]
[New LWP 14400]
[New LWP 14399]
[New LWP 14398]
[New LWP 14397]
[New LWP 14396]
[New LWP 14395]
[New LWP 14394]
[New LWP 14393]
[New LWP 14392]
[New LWP 14391]
[New LWP 14390]
[New LWP 14389]
[New LWP 14388]
[New LWP 14387]
[New LWP 14386]
[New LWP 14385]
[New LWP 14384]
[New LWP 14383]
[New LWP 14382]
[New LWP 14381]
[New LWP 14380]
[New LWP 14379]
[New LWP 14378]
[New LWP 14377]
[New LWP 14376]
[New LWP 14375]
[New LWP 14374]
[New LWP 14373]
[New LWP 14372]
[New LWP 14371]
[New LWP 14370]
[New LWP 14369]
[New LWP 14368]
[New LWP 14363]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f4afbf1ac73 in wait4 () from /lib64/libc.so.6
#0  0x00007f4afbf1ac73 in wait4 () from /lib64/libc.so.6
#1  0x00007f4b0557804b in ggml_print_backtrace () from /home/sebastiaan/src/llama.cpp/build/bin/libggml-base.so.0
#2  0x00007f4b055781a6 in ggml_abort () from /home/sebastiaan/src/llama.cpp/build/bin/libggml-base.so.0
#3  0x00007f4b0259f653 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /home/sebastiaan/src/llama.cpp/build/bin/libggml-cuda.so.0
#4  0x00007f4b028590a3 in void launch_fattn<512, 16, 4>(ggml_backend_cuda_context&, ggml_tensor*, void (*)(char const*, char const*, char const*, char const*, char const*, int const*, float*, float2*, float, float, float, float, unsigned int, float, int, uint3, int, int
, int, int, int, int, int, int, int, int, int, long, int, int, long, int, int, int, int, int, long), int, unsigned long, int, bool, bool, bool, int) () from /home/sebastiaan/src/llama.cpp/build/bin/libggml-cuda.so.0
#5  0x00007f4b02859433 in void ggml_cuda_flash_attn_ext_mma_f16_case<576, 512, 16, 4>(ggml_backend_cuda_context&, ggml_tensor*) () from /home/sebastiaan/src/llama.cpp/build/bin/libggml-cuda.so.0
#6  0x00007f4b025b19c4 in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool) () from /home/sebastiaan/src/llama.cpp/build/bin/libggml-cuda.so.0
#7  0x00007f4b025b2d7e in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/sebastiaan/src/llama.cpp/build/bin/libggml-cuda.so.0
#8  0x00007f4b05593dd6 in ggml_backend_sched_graph_compute_async () from /home/sebastiaan/src/llama.cpp/build/bin/libggml-base.so.0
#9  0x00007f4b04cac751 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/sebastiaan/src/llama.cpp/build/bin/libllama.so.0
#10 0x00007f4b04cae404 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/sebastiaan/src/llama.cpp/build/bin/libllama.so.0
#11 0x00007f4b04cb5661 in llama_context::decode(llama_batch const&) () from /home/sebastiaan/src/llama.cpp/build/bin/libllama.so.0
#12 0x00007f4b04cb701b in llama_decode () from /home/sebastiaan/src/llama.cpp/build/bin/libllama.so.0
#13 0x0000000000542303 in server_context_impl::update_slots() ()
#14 0x0000000000588fdf in server_queue::start_loop(long) ()
#15 0x00000000004a781d in main ()
[Inferior 1 (process 14362) detached]
Aborted (core dumped)

[MannerMan]

I also experienced a crash at ~60k+ context size

[...]
slot update_slots: id  0 | task 4560 | prompt processing progress, n_tokens = 58778, batch.n_tokens = 2048, progress = 0.779984
slot update_slots: id  0 | task 4560 | n_tokens = 58778, memory_seq_rm [58778, end)
slot update_slots: id  0 | task 4560 | prompt processing progress, n_tokens = 60826, batch.n_tokens = 2048, progress = 0.807160
srv          stop: cancel task, id_task = 4560
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
srv  params_from_: Chat format: GLM 4.5
slot      release: id  0 | task 4560 | stop processing: n_tokens = 60826, truncated = 0
slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.807 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ?top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  0 | task 4569 | processing task, is_child = 0
slot update_slots: id  0 | task 4569 | new prompt, n_ctx_slot = 80128, n_keep = 0, task.n_tokens = 75358
slot update_slots: id  0 | task 4569 | n_tokens = 60826, memory_seq_rm [60826, end)
slot update_slots: id  0 | task 4569 | prompt processing progress, n_tokens = 62874, batch.n_tokens = 2048, progress = 0.834337
slot update_slots: id  0 | task 4569 | n_tokens = 62874, memory_seq_rm [62874, end)
slot update_slots: id  0 | task 4569 | prompt processing progress, n_tokens = 64922, batch.n_tokens = 2048, progress = 0.861514
slot update_slots: id  0 | task 4569 | n_tokens = 64922, memory_seq_rm [64922, end)
slot update_slots: id  0 | task 4569 | prompt processing progress, n_tokens = 66970, batch.n_tokens = 2048, progress = 0.888691
/app/ggml/src/ggml-cuda/ggml-cuda.cu:96: CUDA error
CUDA error: invalid configuration argument
  current device: 0, in function launch_fattn at /app/ggml/src/ggml-cuda/template-instances/../fattn-common.cuh:1004
  cudaGetLastError()
libggml-base.so.0(+0x1848b)[0x7f4783b7648b]
libggml-base.so.0(ggml_print_backtrace+0x21f)[0x7f4783b768ef]
libggml-base.so.0(ggml_abort+0x152)[0x7f4783b76ac2]
/app/libggml-cuda.so(+0x166036)[0x7f4768af2036]
/app/libggml-cuda.so(_Z12launch_fattnILi512ELi16ELi4EEvR25ggml_backend_cuda_contextP11ggml_tensorPFvPKcS5_S5_S5_S5_PKiPfP6float2ffffjfi5uint3iiiiiiiiiiiliiliiiiilEimibbbi+0x15dc)[0x7f4768d8afdc]
/app/libggml-cuda.so(_Z37ggml_cuda_flash_attn_ext_mma_f16_caseILi576ELi512ELi16ELi4EEvR25ggml_backend_cuda_contextP11ggml_tensor+0x204)[0x7f4768d8b5a4]
/app/libggml-cuda.so(+0x17678f)[0x7f4768b0278f]
/app/libggml-cuda.so(+0x179872)[0x7f4768b05872]
/app/libggml-cuda.so(+0x17af13)[0x7f4768b06f13]
libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x81f)[0x7f4783b93a9f]
libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1)[0x7f4783cca7c1]
libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x115)[0x7f4783ccac05]
libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x413)[0x7f4783cd2ec3]
libllama.so.0(llama_decode+0x10)[0x7f4783cd4330]
/app/llama-server(+0x13c382)[0x56449c4ee382]
/app/llama-server(+0x177ca6)[0x56449c529ca6]
/app/llama-server(+0x9a5ce)[0x56449c44c5ce]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f478362bd90]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f478362be40]
/app/llama-server(+0x9db65)[0x56449c44fb65]
ggml_cuda_init: found 3 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
  Device 1: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
  Device 2: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
build: 7814 (a3e812811) with GNU 11.4.0 for Linux x86_64
system info: n_threads = 6, n_threads_batch = 6, total_threads = 6

system_info: n_threads = 6 (n_threads_batch = 6) / 6 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

Running with:

          /app/llama-server
          --port 9503
          --metrics --cache-type-k q8_0
          --slots
          --no-mmap
          --flash-attn on
          --model /models/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q6_K_XL.gguf
          --threads 6
          --ctx-size 80000
          --parallel 1
          --tensor-split 14,14,14
          --temp 0.7
          --top-p 1.0
          --min-p 0.01
          --repeat-penalty 1.0
          --fit off
          --jinja

llama b7814 release

[simcop2387]

Just ran into this myself after testing this morning on an A6000 48GB. I've got more of a backtrace but i don't know how useful it is since i'm custom building things myself. I can probably dig into the build artifacts and dig up anything from it if needed though.

slot update_slots: id  1 | task 0 | n_tokens = 63488, memory_seq_rm [63488, end)
slot update_slots: id  1 | task 0 | prompt processing progress, n_tokens = 65536, batch.n_tokens = 2048, progress = 0.561803
/build/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:96: CUDA error
CUDA error: invalid configuration argument
  current device: 0, in function launch_fattn at /build/llama.cpp/ggml/src/ggml-cuda/template-instances/../fattn-common.cuh:1004
  cudaGetLastError()
/build/llama.cpp/build/bin/libggml-base.so.0(+0x1825b)[0x7f36d0e3c25b]
/build/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x21c)[0x7f36d0e3c6bc]
/build/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x15b)[0x7f36d0e3c89b]
/app/libggml-cuda.so(_Z15ggml_cuda_errorPKcS0_S0_iS0_+0xb7)[0x7f36bf4f1d77]
/app/libggml-cuda.so(_Z12launch_fattnILi512ELi16ELi4EEvR25ggml_backend_cuda_contextP11ggml_tensorPFvPKcS5_S5_S5_S5_PKiPfP6float2ffffjfi5uint3iiiiiiiiiiiliiliiiiilEimibbbi+0x164e)[0x7f36bf78ad7e]
/app/libggml-cuda.so(_Z37ggml_cuda_flash_attn_ext_mma_f16_caseILi576ELi512ELi16ELi4EEvR25ggml_backend_cuda_contextP11ggml_tensor+0x1f3)[0x7f36bf78b503]
/app/libggml-cuda.so(+0x19448e)[0x7f36bf50448e]
/app/libggml-cuda.so(+0x1984b3)[0x7f36bf5084b3]
/app/libggml-cuda.so(+0x19a2c3)[0x7f36bf50a2c3]
/build/llama.cpp/build/bin/libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x817)[0x7f36d0e58e07]
/build/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1)[0x7f36d0f97a01]
/build/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x114)[0x7f36d0f994c4]
/build/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x435)[0x7f36d0fa0c65]
/build/llama.cpp/build/bin/libllama.so.0(llama_decode+0xf)[0x7f36d0fa250f]
/app/llama-server(+0x14cb98)[0x557736950b98]
/app/llama-server(+0x192e2e)[0x557736996e2e]
/app/llama-server(+0xae220)[0x5577368b2220]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7f36d08a71ca]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7f36d08a728b]
/app/llama-server(+0xb2785)[0x5577368b6785]

[pwilkin]

@JohannesGaessler @am17an looks like a block size overflow for some kernel maybe?

[Cisien]

I started hitting this today as well. The crash happens right at the point where i would expect the prompt to start streaming.

docker run \
  -it \
  --gpus=all \
  --rm \
  --entrypoint /app/llama-server \
  -p 0.0.0.0:8080:8080 \
  -v ~/.cache:/root/.cache \
  ghcr.io/ggml-org/llama.cpp:full-cuda \
    --host 0.0.0.0 \
    --parallel 1 \
    -hf unsloth/GLM-4.7-Flash-GGUF:Q6_K_XL \
    --jinja \
    -c 202752 \
    --fit on \
    --temp 0.7 \
    --min-p 0.01 \
    --top-p 1.0 \
    --alias incompetent-robot \
    -ngl 99 \
    --flash-attn on \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --verbose

the docker image was updated earlier today: ghcr.io/ggml-org/llama.cpp:full-cuda 3af49db6a68a

que    start_loop: processing task, id = 511
que    start_loop: update slots
srv  update_slots: posting NEXT_RESPONSE
que          post: new task, id = 512, front = 0
slot update_slots: id  0 | task 2 | n_tokens = 63491, memory_seq_rm [63491, end)
slot update_slots: id  0 | task 2 | prompt processing progress, n_tokens = 65539, batch.n_tokens = 2048, progress = 0.957515
srv  update_slots: decoding batch, n_tokens = 2048
clear_adapter_lora: call
set_embeddings: value = 0
/app/ggml/src/ggml-cuda/ggml-cuda.cu:96: CUDA error
CUDA error: invalid configuration argument
  current device: 0, in function launch_fattn at /app/ggml/src/ggml-cuda/template-instances/../fattn-common.cuh:1004
  cudaGetLastError()
libggml-base.so.0(+0x1848b)[0x70c0bc5e248b]
libggml-base.so.0(ggml_print_backtrace+0x21f)[0x70c0bc5e28ef]
libggml-base.so.0(ggml_abort+0x152)[0x70c0bc5e2ac2]
/app/libggml-cuda.so(+0x166036)[0x70c0a155e036]
/app/libggml-cuda.so(_Z12launch_fattnILi512ELi16ELi4EEvR25ggml_backend_cuda_contextP11ggml_tensorPFvPKcS5_S5_S5_S5_PKiPfP6float2ffffjfi5uint3iiiiiiiiiiiliiliiiiilEimibbbi+0x15dc)[0x70c0a17f6fdc]
/app/libggml-cuda.so(_Z37ggml_cuda_flash_attn_ext_mma_f16_caseILi576ELi512ELi16ELi4EEvR25ggml_backend_cuda_contextP11ggml_tensor+0x204)[0x70c0a17f75a4]
/app/libggml-cuda.so(+0x17678f)[0x70c0a156e78f]
/app/libggml-cuda.so(+0x179872)[0x70c0a1571872]
/app/libggml-cuda.so(+0x17af13)[0x70c0a1572f13]
libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x81f)[0x70c0bc5ffa9f]
libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1)[0x70c0bc7367c1]
libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x115)[0x70c0bc736c05]
libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x413)[0x70c0bc73eec3]
libllama.so.0(llama_decode+0x10)[0x70c0bc740330]
/app/llama-server(+0x13c382)[0x62ee27448382]
/app/llama-server(+0x177ca6)[0x62ee27483ca6]
/app/llama-server(+0x9a5ce)[0x62ee273a65ce]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x70c0bc097d90]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x70c0bc097e40]
/app/llama-server(+0x9db65)[0x62ee273a9b65]

[Cisien]

GLM-4.7-Flash seems to think that the fallback should call ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<DKQ, DV, 2>(ctx, dst); instead of ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<DKQ, DV, 1>(ctx, dst); on ggml/src/ggml-cuda/fattn.cu

Metadata is CORRECT:
- head_count = 20
- head_count_kv = 1
- GQA ratio = 20 (valid but edge case)

The Bug: Kernel selection logic should select ncols1=4 for gqa_ratio=20 (because 20 % 4 = 0), but ncols1=2 is being selected instead.

The Fix: Change line 78 in fattn.cu from ncols1=1 to ncols1=2:
// Line 78 - current (WRONG):
ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<DKQ, DV, 1>(ctx, dst);

// Line 78 - fix (CORRECT):
ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<DKQ, DV, 2>(ctx, dst);

This ensures non-power-of-2 GQA ratios use ncols1=2 as a fallback instead of ncols1=1, which violates kernel assumptions.

I don't know enough about llama.cpp/cuda to know if this is just slop, or if the incompetent robot is pointing me in the right direction.

[kkaarrss]

Here as well running into this consistently. Just pulled and build the latest version but issue still exists. After around 60k tokens it crashes. Tried the q6 and q8 unsloth quants but both have the same problem.

slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.842 (> 0.100 thold), f_keep = 0.994
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 7488 | processing task, is_child = 0
slot update_slots: id 3 | task 7488 | new prompt, n_ctx_slot = 75008, n_keep = 0, task.n_tokens = 67182
slot update_slots: id 3 | task 7488 | n_tokens = 56585, memory_seq_rm [56585, end)
slot update_slots: id 3 | task 7488 | prompt processing progress, n_tokens = 58633, batch.n_tokens = 2048, progress = 0.872749
slot update_slots: id 3 | task 7488 | n_tokens = 58633, memory_seq_rm [58633, end)
slot update_slots: id 3 | task 7488 | prompt processing progress, n_tokens = 60681, batch.n_tokens = 2048, progress = 0.903233
slot update_slots: id 3 | task 7488 | n_tokens = 60681, memory_seq_rm [60681, end)
slot update_slots: id 3 | task 7488 | prompt processing progress, n_tokens = 62729, batch.n_tokens = 2048, progress = 0.933717
/home/pluskars/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:96: CUDA error
CUDA error: invalid configuration argument
current device: 0, in function launch_fattn at /home/pluskars/llama.cpp/ggml/src/ggml-cuda/template-instances/../fattn-common.cuh:1004
cudaGetLastError()
[New LWP 750860]
[New LWP 750868]
[New LWP 750869]
[New LWP 750870]
[New LWP 750871]
[New LWP 750872]
[New LWP 750873]
[New LWP 750874]
[New LWP 750875]
[New LWP 750876]
[New LWP 750877]
[New LWP 750878]
[New LWP 750879]
[New LWP 750880]
[New LWP 750881]
[New LWP 750882]
[New LWP 750883]
[New LWP 750884]
[New LWP 750885]
[New LWP 750886]
[New LWP 750887]
[New LWP 750888]
[New LWP 750889]
[New LWP 750890]
[New LWP 750891]
[New LWP 750892]
[New LWP 750893]
[New LWP 750894]
[New LWP 750895]
[New LWP 750896]
[New LWP 750897]
[New LWP 750898]
[New LWP 750899]
[New LWP 750900]
[New LWP 750901]
[New LWP 750902]
[New LWP 750903]
[New LWP 750904]
[New LWP 750905]
[New LWP 750906]
[New LWP 750907]
[New LWP 750908]
[New LWP 750909]
[New LWP 750910]
[New LWP 750911]
[New LWP 750912]
[New LWP 750913]
[New LWP 750914]
[New LWP 750915]
[New LWP 750916]
[New LWP 750917]
[New LWP 750935]
[New LWP 750948]
[New LWP 750949]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f23670fa42f in __GI___wait4 (pid=781127, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
#0 0x00007f23670fa42f in __GI___wait4 (pid=781127, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 in ../sysdeps/unix/sysv/linux/wait4.c
#1 0x00007f236759e92b in ggml_print_backtrace () from libggml-base.so.0
#2 0x00007f236759eac2 in ggml_abort () from libggml-base.so.0
#3 0x00007f235c9778c6 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from libggml-cuda.so.0
#4 0x00007f235cc1016c in void launch_fattn<512, 16, 4>(ggml_backend_cuda_context&, ggml_tensor*, void ()(char const, char const*, char const*, char const*, char const*, int const*, float*, float2*, float, float, float, float, unsigned int, float, int, uint3, int, int, int, int, int, int, int, int, int, int, int, long, int, int, long, int, int, int, int, int, long), int, unsigned long, int, bool, bool, bool, int) () from libggml-cuda.so.0
#5 0x00007f235cc1071d in void ggml_cuda_flash_attn_ext_mma_f16_case<576, 512, 16, 4>(ggml_backend_cuda_context&, ggml_tensor*) () from libggml-cuda.so.0
#6 0x00007f235c987dcf in ggml_cuda_compute_forward(ggml_backend_cuda_context&, ggml_tensor*) () from libggml-cuda.so.0
#7 0x00007f235c98b2c2 in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) () from libggml-cuda.so.0
#8 0x00007f235c98cc9b in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from libggml-cuda.so.0
#9 0x00007f23675bba9f in ggml_backend_sched_graph_compute_async () from libggml-base.so.0
#10 0x00007f23676f07b1 in llama_context::graph_compute(ggml_cgraph*, bool) () from libllama.so.0
#11 0x00007f23676f0bf5 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from libllama.so.0
#12 0x00007f23676f8eb3 in llama_context::decode(llama_batch const&) () from libllama.so.0
#13 0x00007f23676fa320 in llama_decode () from libllama.so.0
#14 0x000055fedba7db92 in server_context_impl::update_slots() ()
#15 0x000055fedbab94b6 in server_queue::start_loop(long) ()
#16 0x000055fedb9dbebe in main ()
[Inferior 1 (process 750859) detached]

Running on 2x 3090 on Ubuntu natively.
CUDA_VISIBLE_DEVICES=1,2 nohup ./llama-server --host 0.0.0.0 -ngl 99999 --model /mnt/ssd/models/GLM-4.7-Flash-UD-Q8_K_XL.gguf -ctv q8_0 -ctk q8_0 --flash-attn on --temp 0.7 --top-p 1.0 --min-p 0.01 --jinja --port 8002 --no-mmap -c 64000 --cache-ram 0 > glm_flash.log 2>&1 &

[ggerganov]

I haven't looked in details what happens with quantized KV cache, but I suspect the issue here is that you are running out of memory because quantized KV cache requires some extra buffers during compute to kast the K tensor to F16 and this brings you over the edge. Try disabling auto memory fit and use a smaller context size in order to leave some room for the dequantized K cache.

Also, consider just not using quantized KV cache - in most cases it's not worth it.

[JohannesGaessler]

The problem was introduced with #18986 because it made the V tensor non-contiguous which causes issues on dequantization for >= 65536 tokens. This should already be fixed on the latest master version by #19057 . Anyone having problems after that PR should make a new Github issue and completely fill out the "model use" template.

[SharkWipf]

Thanks, should have figured something like that was up when it was only happening at >~60k token inputs.
I have confirmed it works, seems to be working great now.

Also, consider just not using quantized KV cache - in most cases it's not worth it.

I'm going to have to respectfully but strongly disagree here, for those of us who are GPU poor, KV quantization can be the difference between 100t/s and <10t/s, or even between being able to run a model at all, at a in most cases negligible reduction in generation quality where it matters (workload-dependent of course).
In this specific case, with this model, with KV quant the entire model (barely) fits in 24GB VRAM at full 200k context, which in my case is more important than the quality of its output (which is still excellent for my purposes even at q4_1), while getting a fairly respectable ~25t/s at this size. Without KV quant, I have to rely on --no-kv-offload or partial offloading, both reducing speeds to only a fraction of what I'm getting here.

While I'm sure it reduces quality in some scenarios, KV quant has been pretty close to a free lunch for me, across various models.

[am17an]

@ggerganov while we are on the topic. Using fp8 is known to have a near lossless quality when quantising. I'm not familiar with how q8 behaves, but is this a direction you will consider? For FA it will be useful as we accumulate in f32 while using fp8 activations using native hardware.