FA bug causes Memory access fault by GPU

[cb88]

Name and Version

./bin/llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
Device 1: AMD Instinct MI60 / MI50, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
version: 4819 (becade5)
built with cc (GCC) 14.2.1 20250207 for x86_64-pc-linux-gnu

Operating systems

Linux

GGML backends

HIP

Hardware

AMD EPYC 7352 (48) @ 2.300GHz
GPU: AMD Radeon Instinct MI60 32GB
GPU: AMD Radeon Instinct MI50 32GB
Memory: 128669MiB

Models

llama-2-7b.Q4_0.gguf

Problem description & steps to reproduce

run llama-bench with -fa 1

First Bad Commit

Git bisected to becade5

Relevant log output

./bin/llama-bench  -m ~/Downloads/llama-2-7b.Q4_0.gguf  -fa 1 -sm none  -mg 0
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 1: AMD Instinct MI60 / MI50, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |    sm | fa |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ----: | -: | ------------: | -------------------: |
Memory access fault by GPU node-1 (Agent handle: 0x5cf36b0ef290) on address 0x7ba863e00000. Reason: Page not present or supervisor privilege.

[IMbackK]

Did you Compiler with GGML_HIP_ROCWMMA_FATTN=On? This is a bit misterious as that commit should not have changed the codepath for gfx906 at all.

[cb88]

I have never set that flag. So nope. Anything I can check to see if it is somehow getting enabled unintentionally?

[cb88]

I built it with these flags,
cmake .. -DGGML_CUDA_FORCE_MMQ=OFF -DGGML_CUDA_FORCE_CUBLAS=OFF -DGGML_HIP=ON -DCMAKE_BUILD_TYPE=Release -DAMDGPU_TARGETS=gfx906

also checked and adding -DGGML_HIP_ROCWMMA_FATTN=On amd GGML_HIP_ROCWMMA_FATTN=OFF at compile time do not change the results.

[unclemusclez]

I also have this same issue with MI60 GFX906 on the tags b4848+. The earliest working version I'm aware of is sometime after the Deepseek compatibility addition.

this occurred on both ROCm 6.3.1 and 6.3.3 Ubuntu 24.04.

[unclemusclez]

@cb88 do you know which card caused the issue? i also have the mi60. it seems like only only of your devices caused the issue. i'm guessing the one that says ".../MI60..." is the card that caused the problem.

this may be an AMD issue.

[unclemusclez]

ive reverted to b4818, the release prior to the WMMA FA update and it works.

[IMbackK]

yeah very strage fiels like a compiler bug looking at the diff, but I will try to find the time to try and repo it with mi100 by making it follow the mi50 code path

[cb88]

@cb88 do you know which card caused the issue? i also have the mi60. it seems like only only of your devices caused the issue. i'm guessing the one that says ".../MI60..." is the card that caused the problem.

this may be an AMD issue.

No it doesn't matter which one I run it against.

[IMbackK]

I figured this out, and i just missed an obvious thing on first reading: the commit from the bisect changes the dprod functions to work on 64 wide waves but the vec fatten kernels only allocate 32 wide buffers, so ofc this crashes. I will write a fix patch soon.

Any non wave32 device that dosent also have mfma should be broken.