Eval bug: HIP + turbo3_tcq = crash

[Ezzz-dev]

Name and Version

latest

Operating systems

Linux

GGML backends

HIP

Hardware

7900XTX GFX1100

Models

Qwen 3.6 27B Q4KM

Problem description & steps to reproduce

Normal llama run with 128k context size, turbo3_tcp modes

First Bad Commit

No response

Relevant log output

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f8f98280e32 in __syscall_cancel_arch () from /lib64/libc.so.6
#0 0x00007f8f98280e32 in __syscall_cancel_arch () from /lib64/libc.so.6
#1 0x00007f8f9827506c in __internal_syscall_cancel () from /lib64/libc.so.6
#2 0x00007f8f982750b4 in __syscall_cancel () from /lib64/libc.so.6
#3 0x00007f8f982e552f in wait4 () from /lib64/libc.so.6
#4 0x00007f8f98f1dd9b in ggml_print_backtrace () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-base.so.0
#5 0x00007f8f98f1df0d in ggml_abort () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-base.so.0
#6 0x00007f8f9e39b582 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#7 0x00007f8f9e56d127 in void launch_fattn<256, 2, 2>(ggml_backend_cuda_context&, ggml_tensor*, void ()(char const, char const*, char const*, char const*, char const*, int const*, float*, HIP_vector_type<float, 2u>, float, float, float, float, unsigned int, float, int, HIP_vector_type<unsigned int, 3u>, int, int, int, int, int, int, int, int, int, int, int, long, int, int, long, int, int, int, int, int, long), int, unsigned long, int, bool, bool, bool, int) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#8 0x00007f8f9e557b55 in void ggml_cuda_flash_attn_ext_tile_case<256, 256>(ggml_backend_cuda_context&, ggml_tensor) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#9 0x00007f8f9e37b70d in ggml_cuda_flash_attn_ext(ggml_backend_cuda_context&, ggml_tensor*) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#10 0x00007f8f9e3a5afe in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#11 0x00007f8f9e3a1fdc in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#12 0x00007f8f98f3a83b in ggml_backend_sched_graph_compute_async () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-base.so.0
#13 0x00007f8f9ee4cfb0 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libllama.so.0
#14 0x00007f8f9ee5119b in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libllama.so.0
#15 0x00007f8f9ee59d9f in llama_context::decode(llama_batch const&) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libllama.so.0
#16 0x00007f8f9ee5c85e in llama_decode () from /home/ezzz/Desktop/beellama.cpp/build/bin/libllama.so.0
#17 0x00007f8f9f5494a2 in common_context_can_seq_rm(llama_context*) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libllama-common.so.0
spiritbuun#18 0x00000000004f861e in server_context_impl::load_model(common_params&) ()
spiritbuun#19 0x000000000043a0a7 in main ()
[Inferior 1 (process 19251) detached]

[Anbeeld]

I pushed a diagnostic branch for this issue: hip-turbo3-tcq-crash.

This branch does not try to fix the crash yet. It adds logging around the HIP flash-attention path so we can tell whether the failure is coming from turbo K/V dequant, Q rotation, or the final TILE dispatch.

Could you please build this branch and run the same command that currently crashes?

First, run the baseline from the branch:

./your-command-here

Then run with diagnostics:

GGML_FATTN_DEBUG=1 GGML_FATTN_DEBUG_SYNC=1 ./your-command-here

Please paste the last 30–50 lines of output around the failure. The most useful line will look like one of these:

FATTN launch error after ...
FATTN sync error after ...
FATTN select: ...

After that, please try forcing VEC:

GGML_FATTN_DEBUG=1 GGML_FATTN_FORCE_KERNEL=vec ./your-command-here

And forcing TILE:

GGML_FATTN_DEBUG=1 GGML_FATTN_FORCE_KERNEL=tile ./your-command-here

One more useful comparison is native turbo decode, which skips the fp16 dequant path:

GGML_TURBO_DECODE_NATIVE=1 ./your-command-here

If you can, also test plain fp16 KV cache with forced TILE:

GGML_FATTN_DEBUG=1 GGML_FATTN_FORCE_KERNEL=tile ./your-command-here --cache-type-k f16 --cache-type-v f16

That last run helps separate "TILE is broken for D=256 on HIP in general" from "TILE only fails after turbo3_tcq dequant creates the temporary fp16 K/V layout."

Please include:

GPU:
ROCm version:
Build flags:
Model:
Full command:
Result of baseline:
Result with GGML_FATTN_DEBUG=1 GGML_FATTN_DEBUG_SYNC=1:
Result with GGML_FATTN_FORCE_KERNEL=vec:
Result with GGML_FATTN_FORCE_KERNEL=tile:
Result with GGML_TURBO_DECODE_NATIVE=1:
Result with f16/f16 KV + forced TILE:

If GGML_FATTN_DEBUG_SYNC=1 conflicts with HIP graph capture, rebuild with graphs disabled, for example with -DGGML_HIP_GRAPHS=OFF, then rerun the diagnostic command.

Once we have those logs, I should be able to tell whether the real fix belongs in the dequant kernels, the Q rotation step, the TILE selector/config, or the TILE handling of the temporary fp16 K/V strides.