Eval bug: unsloth/Kimi-K2-Thinking-GGUF:UD-Q4_K_XL -ot ".ffn_.*_exps.=CPU"

[thread3d]

Name and Version

./llama.cpp/build/bin/llama-cli --version
ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV NAVI21) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon Graphics (RADV NAVI21) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
ggml_vulkan: 2 = AMD Radeon Graphics (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: none
version: 7058 (45c6ef7)
built with cc (GCC) 15.2.1 20251022 (Red Hat 15.2.1-3) for x86_64-redhat-linux

Operating systems

Linux

GGML backends

Vulkan

Hardware

AMD Radeon Pro Duo W6800X + AMD Radeon Pro Vega II
No infinity fabric link installed

Models

unsloth/Kimi-K2-Thinking-GGUF:UD-TQ1_0
unsloth/Kimi-K2-Thinking-GGUF:UD-Q4_K_XL

Problem description & steps to reproduce

./build/bin/llama-cli -hf unsloth/Kimi-K2-Thinking-GGUF:UD-Q4_K_XL -ot ".ffn_.*_exps.=CPU" --special

"Tell me all about the rise and fall of the Roman Empire"

After only a few tokens, or a few hundred, it crashes with a seg fault.

Sometime later, the machine will abend and restart.

First Bad Commit

No response

Relevant log output

This GDB supports auto-downloading debuginfo from the following URLs:
  <ima:enforcing>
  <https://debuginfod.fedoraproject.org/>
  <ima:ignore>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fe5fc286982 in __syscall_cancel_arch () from /lib64/libc.so.6
#0  0x00007fe5fc286982 in __syscall_cancel_arch () from /lib64/libc.so.6
#1  0x00007fe5fc27ac3c in __internal_syscall_cancel () from /lib64/libc.so.6
#2  0x00007fe5fc27ac84 in __syscall_cancel () from /lib64/libc.so.6
#3  0x00007fe5fc2eab4f in wait4 () from /lib64/libc.so.6
#4  0x00007fe5ffd02963 in ggml_print_backtrace () from /home/threaded/projects/llama.cpp/build/bin/libggml-base.so.0
#5  0x00007fe5ffd02aaf in ggml_abort () from /home/threaded/projects/llama.cpp/build/bin/libggml-base.so.0
#6  0x00007fe5fc8df839 in ggml_vk_mul_mat_vec_q_f16(ggml_backend_vk_context*, std::shared_ptr<vk_context_struct>&, ggml_cgraph const*, int) () from /home/threaded/projects/llama.cpp/build/bin/libggml-vulkan.so.0
#7  0x00007fe5fc90d1f0 in ggml_vk_build_graph(ggml_backend_vk_context*, ggml_cgraph*, int, ggml_tensor*, int, bool, bool, bool) () from /home/threaded/projects/llama.cpp/build/bin/libggml-vulkan.so.0
#8  0x00007fe5fc90f862 in ggml_backend_vk_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/threaded/projects/llama.cpp/build/bin/libggml-vulkan.so.0
#9  0x00007fe5ffd1cde3 in ggml_backend_sched_graph_compute_async () from /home/threaded/projects/llama.cpp/build/bin/libggml-base.so.0
#10 0x00007fe5ffa384d0 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/threaded/projects/llama.cpp/build/bin/libllama.so.0
#11 0x00007fe5ffa3a162 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/threaded/projects/llama.cpp/build/bin/libllama.so.0
#12 0x00007fe5ffa3f03f in llama_context::decode(llama_batch const&) () from /home/threaded/projects/llama.cpp/build/bin/libllama.so.0
#13 0x00007fe5ffa3fe8e in llama_decode () from /home/threaded/projects/llama.cpp/build/bin/libllama.so.0
#14 0x000000000042f2b0 in main ()
[Inferior 1 (process 15299) detached]
Aborted                    (core dumped) ./llama.cpp/build/bin/llama-cli -hf unsloth/Kimi-K2-Thinking-GGUF:UD-Q4_K_XL -ot ".ffn_.*_exps.=CPU" --special

[jeffbolznv]

I tried to reproduce this with the TQ1_0 model, but haven't been able to so far. Can you catch which line is causing the ggml_abort?

[thread3d]

I've built a debug version, and the line numbers are shown.
run_1.log

[jeffbolznv]

Based on the assert, looks like this might be an interaction between the q8_1 path and noncontiguous src1. @0cc4m do you know whether we have test coverage for that case?

[0cc4m]

Based on the assert, looks like this might be an interaction between the q8_1 path and noncontiguous src1. @0cc4m do you know whether we have test coverage for that case?

I don't think so, since quantize_y requires contiguous src1. The assertion is really old, so it may not be correct anymore. But on first glance I can't see a reason why it wouldn't go through. Maybe we need a GGML_VULKAN_DEBUG run?

[thread3d]

With GGML_VULKAN_DEBUG added
run_1.log

[0cc4m]

Thank you, that is actually very helpful.

So the problematic tensor is this:

(0x1df3400, name=__fattn__-42 (permuted), type=0, ne0=512, ne1=1, ne2=64, ne3=1, nb0=4, nb1=131072, nb2=2048, nb3=131072)

It's clearly permuted (even named as such), but because ne1==1 ggml_is_contiguous does not check that dimension and returns true. That does not seem right to me. The description of the function also states no gaps, no permutation.

@slaren is this expected behaviour from ggml_is_contiguous?

[slaren]

I don't think calling this tensor contiguous is wrong. Permutations of dimensions of size 1 do not change the layout of the data in memory, so I don't think this should be an issue.

[0cc4m]

I don't think calling this tensor contiguous is wrong. Permutations of dimensions of size 1 do not change the layout of the data in memory, so I don't think this should be an issue.

It's an issue for me in this case, since I do expect no permutation. We also have ggml_is_contiguously_allocated for cases where permutation is allowed.

Either the description is wrong and the second function is superfluous, or the implementation is wrong. The description and second function came from @JohannesGaessler, so it seems like he understood it the same way I did.

[slaren]

Why is this an issue for Vulkan? To reduce this to the simplest case, you could permute a tensor of dimensions [1,1,1,1], and the end result will be the same regardless. Does it actually cause issues if you remove the assert?

The problems seems to me that ggml_vk_dim01_contiguous is using a naive implementation that is not able to tell accurately if a tensor is contiguous, and it results in a mismatch with the result of ggml_is_contiguous.

[0cc4m]

In this case, because the matrix multiplication shaders expect the first two dimensions to be contiguous, to simplify the index offset calculations. If they are not, a separate shader is called first to copy the tensor into a contiguous, non-permuted shape. This is checked with ggml_vk_dim01_contiguous. For this tensor, that function returns false, while ggml_is_contiguous returns true, which triggers activation quantization (which also assumes no permutation).

Of course I can fix it in the backend, but let's clear up the GGML API functions first.

[slaren]

The implementation of ggml_vk_dim01_contiguous already has an exception where it ignores the value of nb3 if ne3 == 1. This comes from #14683, which correctly diagnosed that the function was not able to detect contiguous tensors in some situations. You just need to extend the logic to the other dimensions.

[JohannesGaessler]

@0cc4m I don't see any contradiction between my understanding of the functionality and Diego's understanding of the functionality. The comment I added refers to the memory layout with "no permutation" (though this is ambiguous). I specifically added the second function to check whether scalar operations can correctly ignore the tensor shape.

[0cc4m]

Okay, thank you, then the issue was with how I understood the function. I'll add a permutation check in the relevant places in the backend to fix this bug.