Eval bug: Encounter fatal error whenever message sent to the llama.cpp server

[CCJetWing]

Name and Version

version: 9683 (8086439)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux, Windows

GGML backends

CUDA

Hardware

CUDA0 : NVIDIA GeForce RTX 3090 (24575 MiB, 30790 MiB free)
CUDA1 : NVIDIA GeForce RTX 3090 (24575 MiB, 30707 MiB free)
CPU : AMD Ryzen 9 5900X 12-Core Processor (32045 MiB, 32045 MiB free)

Models

Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

Problem description & steps to reproduce

whenever I input any message, it shows fatal error and then terminated

First Bad Commit

No response

Relevant log output

~/llama.cpp/build/bin/llama-server -m ~/LLM/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --mmproj ~/LLM/mmproj-BF16.gguf --n-cpu-moe 999 -t 12 -b 512 -ub 128 -fa on -ts 1,1 --jinja --port 8000
0.00.074.564 I log_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.074.583 I device_info:
0.00.241.480 I - CUDA0 : NVIDIA GeForce RTX 3090 (24575 MiB, 30790 MiB free)
0.00.422.896 I - CUDA1 : NVIDIA GeForce RTX 3090 (24575 MiB, 30707 MiB free)
0.00.422.925 I - CPU : AMD Ryzen 9 5900X 12-Core Processor (32045 MiB, 32045 MiB free)
0.00.423.032 I system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.423.053 I srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.423.141 I srv init: using 23 threads for HTTP server
0.00.423.532 I srv start: binding port with default address family
0.00.424.863 I srv llama_server: loading model
0.00.424.887 I srv load_model: loading model '/home/ubuntu/LLM/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf'
0.01.017.642 I srv load_model: [mtmd] estimated worst-case memory usage of mmproj is 1134.00 MiB
0.01.017.690 I common_init_result: fitting params to device memory ...
0.01.017.691 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.03.004.042 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
0.10.324.192 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.10.493.775 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
0.10.493.792 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
0.10.493.792 W load_hparams: more info: #16842

0.10.755.402 I srv load_model: loaded multimodal model, '/home/ubuntu/LLM/mmproj-BF16.gguf'
0.10.755.444 I srv load_model: initializing slots, n_slots = 4
0.10.848.845 W srv load_model: speculative decoding will use checkpoints
0.10.848.879 W common_speculative_init: no implementations specified for speculative decoding
0.10.848.880 I slot load_model: id 0 | task -1 | new slot, n_ctx = 262144
0.10.848.888 I slot load_model: id 1 | task -1 | new slot, n_ctx = 262144
0.10.848.888 I slot load_model: id 2 | task -1 | new slot, n_ctx = 262144
0.10.848.889 I slot load_model: id 3 | task -1 | new slot, n_ctx = 262144
0.10.848.905 I srv load_model: prompt cache is enabled, size limit: 8192 MiB
0.10.848.905 I srv load_model: use --cache-ram 0 to disable the prompt cache
0.10.848.906 I srv load_model: for more info see #16391
0.10.848.906 I srv load_model: context checkpoints enabled, max = 32, min spacing = 256
0.10.848.958 I srv init: idle slots will be saved to prompt cache and cleared upon starting a new task
0.10.862.361 I init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

'
0.10.872.375 I srv init: init: chat template, thinking = 1
0.10.872.430 I srv llama_server: model loaded
0.10.872.435 I srv llama_server: server is listening on http://127.0.0.1:8000
0.10.872.443 I srv update_slots: all slots are idle
0.31.095.064 I srv params_from_: Chat format: peg-native
0.31.095.259 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.31.095.272 I srv get_availabl: updating prompt cache
0.31.095.277 I srv load: - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.31.095.281 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 262144 tokens, 8589934592 est)
0.31.095.282 I srv get_availabl: prompt cache update took 0.01 ms
0.31.095.316 I reasoning-budget: activated, budget=2147483647 tokens
0.31.095.317 I reasoning-budget: deactivated (natural end)
0.31.095.329 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
0.31.095.330 I slot process_sing: id 0 | task -1 | saving idle slot to prompt cache
0.31.095.331 I slot prompt_clear: id 0 | task -1 | clearing prompt with 0 tokens
0.31.095.452 I slot process_sing: id 1 | task -1 | saving idle slot to prompt cache
0.31.095.452 I slot prompt_clear: id 1 | task -1 | clearing prompt with 0 tokens
0.31.095.563 I slot process_sing: id 2 | task -1 | saving idle slot to prompt cache
0.31.095.575 I slot prompt_clear: id 2 | task -1 | clearing prompt with 0 tokens
mmq_x_best=0
/home/ubuntu/llama.cpp/ggml/src/ggml-cuda/template-instances/../mmq.cuh:4135: fatal error
/home/ubuntu/llama.cpp/build/bin/libggml-base.so.0(+0x1b846) [0x7833079d1846]
/home/ubuntu/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x20d) [0x7833079d1bfd]
/home/ubuntu/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x166) [0x7833079d1de6]
/home/ubuntu/llama.cpp/build/bin/libggml-cuda.so.0(_Z14mul_mat_q_caseIL9ggml_type8EEvR25ggml_backend_cuda_contextRK8mmq_argsP11CUstream_st+0x205) [0x7832fc23f685]
/home/ubuntu/llama.cpp/build/bin/libggml-cuda.so.0(Z19ggml_cuda_mul_mat_qR25ggml_backend_cuda_contextPK11ggml_tensorS3_S3_PS1+0xb2a) [0x7832fbaf7a7a]
/home/ubuntu/llama.cpp/build/bin/libggml-cuda.so.0(+0x2d8383) [0x7832fbad8383]
/home/ubuntu/llama.cpp/build/bin/libggml-cuda.so.0(+0x2d8d22) [0x7832fbad8d22]
/home/ubuntu/llama.cpp/build/bin/libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x7be) [0x7833079ef7de]
/home/ubuntu/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa0) [0x7833062dd710]
/home/ubuntu/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x119) [0x7833062e17a9]
/home/ubuntu/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x388) [0x7833062e81f8]
/home/ubuntu/llama.cpp/build/bin/libllama.so.0(llama_decode+0x12) [0x7833062e9fd2]
/home/ubuntu/llama.cpp/build/bin/libllama-server-impl.so(_ZN19server_context_impl12update_slotsEv+0x10de) [0x7833075803de]
/home/ubuntu/llama.cpp/build/bin/libllama-server-impl.so(_ZN12server_queue10start_loopEl+0x1fc2) [0x783307620fb2]
/home/ubuntu/llama.cpp/build/bin/libllama-server-impl.so(_Z12llama_serveriPPc+0x369d) [0x7833074d410d]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a601) [0x783306c2a601]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x88) [0x783306c2a718]
/home/ubuntu/llama.cpp/build/bin/llama-server(+0x1325) [0x5be1da426325]
Aborted ~/llama.cpp/build/bin/llama-server -m ~/LLM/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --mmproj ~/LLM/mmproj-BF16.gguf --n-cpu-moe 999 -t 12 -b 512 -ub 128 -fa on -ts 1,1 --jinja --port 8000

[vbooka1]

Try removing " --n-cpu-moe 999"

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.