AMD ROCm support

[reneleonhardt]

https://github.com/huggingface/candle/tree/main/candle-core/src offers 3 backends: CPU, CUDA and Metal.
Notably, AMD ROCm is missing.

candle-kernels "contains CUDA kernels", AMD support (ROCm contains HIP) is only mentioned twice:
https://github.com/huggingface/candle/blob/main/candle-kernels/src/quantized.cu#L2631

#if defined(GGML_USE_HIPBLAS) && defined(__HIP_PLATFORM_AMD__) && (defined(RDNA2) || defined(RDNA3))

https://github.com/huggingface/candle/blob/main/candle-kernels/src/moe/moe_utils.cuh#L172

#ifndef USE_ROCM

Would it be feasible to add a fourth backend for AMD, possibly analogous to the CUDA backend with the help of HIP?
https://github.com/ROCm/rocm-systems/tree/rocm-7.2.0/projects/hip
https://rocm.docs.amd.com/projects/HIP/en/latest/index.html

[zachelnet]

Hi,

I actually tried testing ROCm with candle. After experiencing multiple OOM kills and needing a LOT of patching, it turned out to be very frustrating and painful.

The main issue right now is compiling and running the HIP kernels on specific consumer GPU targets under ROCm 7.x. It's incredibly brittle and prone to memory management failures.

For the moment, I think it makes the most sense to wait for the next major release, ROCm 8.x. As detailed in the Fedora change proposal linked below, AMD is finally moving towards "generic GPU targets" (like gfx11-generic, gfx10-3-generic, etc.). This should fundamentally fix the kernel fragmentation and compatibility issues we are currently fighting with.

See the Fedora article for context:
https://fedoraproject.org/wiki/Changes/ROCm_8.x

[zachelnet]

On top of the driver/kernel instabilities, the Rust ecosystem bindings are currently broken as well. For example, rocm-rs fails to compile on stable Rust due to missing feature gates and trying to pull in nightly-only dependencies at build time. This forces even more manual patching just to get a basic build going, as documented in this issue:
RustNSparks/rocm-rs#20