Does ROCm 5.7 support Radeon 780M (gfx1103)?

[bod2000]

Hi

Does latest ROCm 5.7 support Radeon 780M (gfx1103)? This chip is part of mobile cpu Ryzen 7940HS.
if its not supported - is there any plans to add support of this GPU?

[riverzhou]

Same question.

[hrstoyanov]

Ryzen 7940HS is selling a lot not only for laptops, but mini computers, homelab servers. It is such a shame that AMD does not support it's products, especially on Linux. This question is ignored without an official answer or guidance from AMD for months now, the Ryzen 7940HS has been selling since April 2023!

I am starting to regret my purchase of Ryzen 7940HS ...

[spitzenidee]

At least llama.cpp breaks after compilation on a 7940HS APU (ROCm installed via AMDGPU_INSTALL and that seemed to be successful in the first place):

rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: No such file or directory for GPU arch : gfx1103
 List of available TensileLibrary Files :
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1030.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx940.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx941.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx908.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx906.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx803.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx90a.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1102.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1101.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx942.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx900.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1100.dat"
Aborted (core dumped)

[spitzenidee]

Mhm, just found that simply copying a "likely similar" TensileLibrary into the necessary file seems to have helped (at least for the llama.cpp use case):

sudo cp /opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1102.dat /opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1103.dat

That's probably not the optimal solution, but it works flawlessly, so far.

Did a quick bench of llama.cpp compiled for ROCm (using make LLAMA_HIPBLAS=1) and for CPU (using make):

ROCm (more than "-ngl 32" resulted in gibberish)

> ./main --color --mlock --temp 0.7 --repeat_penalty 1.1 -n -1 -c 0 -b 1024 -m ~/models_gguf/em_german_leo_mistral.Q5_K_M.gguf -t 14 -ngl 32 -p "Wer bist Du? Beschreibe Dich in maximal 200 Worten:"

llama_print_timings:        load time =   10616.17 ms
llama_print_timings:      sample time =      18.43 ms /   149 runs   (    0.12 ms per token,  8083.77 tokens per second)
llama_print_timings: prompt eval time =    9810.28 ms /    21 tokens (  467.16 ms per token,     2.14 tokens per second)
llama_print_timings:        eval time =   10801.36 ms /   148 runs   (   72.98 ms per token,    13.70 tokens per second)
llama_print_timings:       total time =   20718.78 ms

CPU

> ./main --color --mlock --temp 0.7 --repeat_penalty 1.1 -n -1 -c 0 -b 1024 -m ~/models_gguf/em_german_leo_mistral.Q5_K_M.gguf -t 14 -ngl 0 -p "Wer bist Du? Beschreibe Dich in maximal 200 Worten:"

llama_print_timings:        load time =     998.72 ms
llama_print_timings:      sample time =      11.10 ms /   112 runs   (    0.10 ms per token, 10091.00 tokens per second)
llama_print_timings: prompt eval time =     708.25 ms /    21 tokens (   33.73 ms per token,    29.65 tokens per second)
llama_print_timings:        eval time =   10243.40 ms /   111 runs   (   92.28 ms per token,    10.84 tokens per second)
llama_print_timings:       total time =   11056.88 ms

Verdict: prompt eval time sucks on GPU / APU (takes 10-15x longer), but eval time seems to be roughly ~15-25% faster. Depending on response length, total runtime is 50-100% longer on APU than on pure CPU. My takeaway is that using APU is probably worthwhile for large contexts / responses, but not for quick interactive "chats". Offloading prompt eval onto CPU and response generation onto APU / GPU separately in llama.cpp would be a nice optimization feature, I guess (if it's technically feasible).

[shtirlic]

At least llama.cpp breaks after compilation on a 7940HS APU (ROCm installed via AMDGPU_INSTALL and that seemed to be successful in the first place):

rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: No such file or directory for GPU arch : gfx1103
 List of available TensileLibrary Files :
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1030.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx940.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx941.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx908.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx906.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx803.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx90a.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1102.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1101.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx942.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx900.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1100.dat"
Aborted (core dumped)

Try with override GFX version like this before the command:

export HSA_OVERRIDE_GFX_VERSION=11.0.0

[fluidnumerics-joe]

Based on the output you've shared, it looks like rocblas does not ship with TensileLibrary for gfx1103. Not sure why that env variable isn't working, but perhaps try copying or symlinking the gfx1100.dat file to one with gfx1130.dat, e.g.

cp opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1100.dat opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1130.dat

This being said, since your architecture cannot be found, it seems that ROCm 5.7 does not support Radeon 780M. It's also not shown in their documentation for Radeon GPUs

https://rocm.docs.amd.com/en/latest/release/gpu_os_support.html#linux-supported-gpus

[spitzenidee]

The env variable does indeed work, I just didn't know about it before going the brute-force "Copy the missing library" route. I removed said lib again, set "HSA_OVERRIDE_GFX_VERSION=11.0.2" and recompiled llama.cpp with HIPBLAS support.

I do think that ROCm is supported - in principle at least, not officially - for 780M since llama does use VRAM and also is 10-20% faster when offloading layers into VRAM for 7B and 13B models (larger models seem to be slower with offloading to iGPU than CPU-only). However, prompt eval and loading time is much slower on iGPU compared to CPU-only.

[unoexperto]

Folks, could you please advice what I'm missing in my attempt to use ROCm ? Here are details: ROCm/ROCm-docker#123

[jondo]

[Radeon 780M is] also not shown in their documentation for Radeon GPUs

https://rocm.docs.amd.com/en/latest/release/gpu_os_support.html#linux-supported-gpus

The list of supported GPUs (not listing the Radeon 780M) is now at https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html .

[trixirt]

It seems like folks are using AMD's packaging which may not have support gfx1103 compiled in.
Could someone try Fedora's ROCm, either F40 or rawhide ?
I maintain ROCm there and would like to know if this would be a good platform to test with.
gfx1103 is in our default gpu list so it should work, it works on my discrete gfx1103.
Here's our wiki for some more info
Fedora ROCm

[fluidnumerics-joe]

This warning at https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/amdgpu-install.html might be relevant. It appears ROCm does not support integrated graphics.

[trixirt]

d'OH! It is just variable in a list. Added and doing a rebuild now..

[dreirund]

d'OH! It is just variable in a list. Added and doing a rebuild now..

Where exactly? / Where is your build recipe?

[trixirt]

In general each rpm spec file loops over the gpu family building all the gpus within that family. Here is the change to the list that controls the gfx11 family.
Add gfx1103 to the build

Here is where it feeds into building rocBLAS ROCM_GPUS use

[lihaofd]

Is there any plan to support Radeon 780M (gfx1103) in windows?

[craigcabrey]

I have it working on Fedora CoreOS using ollama & a 760m (7640hs):

podman run --rm --env HSA_OVERRIDE_GFX_VERSION=11.0.0 --device /dev/kfd --device /dev/dri -p 11434:11434 --name ollama ollama/ollama:rocm

Note that I have manually set vram to 8G in the bios.

[fcladera]

Could you share some benchmarks of this solution compared to a cpu-only inference?