RFC - enabling ROCM and Vulkan llama.cpp in containers

[Vuks69]

Before I continue hammering away at my sanity the following problems and creating pull requests, I want to have some clarification on how the maintainers would want these issues to be fixed.

ROCM

llama.cpp is unable to build with ROCM support

Only the GPU hardware is passed through the compose files (/dev/kfd, /dev/dri/renderD128 in docker/gpu.amd.yml), but no ROCm software is present. I'm aware this isn't a bug — the backend gate at routes/cookbook_helpers.py:986 only fires the HIP branch when a ROCm toolchain is detected, and the codebase deliberately reports "rocm" only when hipconfig exists (services/hwfit/hardware.py:292-295, routes/cookbook_routes.py:3016-3034). So a container with no ROCm userspace is supposed to fall back to Vulkan.

The problem is that installing the ROCm stack still does not produce a working build — it just unblocks configure far enough to hit the two blockers below.

Solution(s)

apt-get install --no-install-recommends hipcc libhipblas-dev librocblas-dev # +4,5GB

With this, the HIP compiler is found (Clang 17.0.6) but configure still fails — see below.

SELinux blocks /dev/kfd so ROCm cannot enumerate the GPU

On SELinux-enforcing hosts (Fedora/Bazzite/RHEL) the AMD overlay passes /dev/kfd through, but the container domain (container_t) has no rule for the device's label (hsa_device_t), so rocminfo is denied map and no ROCm agent is visible inside the container:

avc: denied { map } for comm="rocminfo" path="/dev/kfd" scontext=system_u:system_r:container_t:s0 tcontext=system_u:object_r:hsa_device_t:s0 tclass=chr_file permissive=0

This is separate from the HIP version gate below and would block a correctly built ROCm binary too. The Vulkan path is unaffected because /dev/dri/renderD128 is dri_device_t, which the container policy does allow.

Solution(s)

The ROCm and Red Hat guidance for AMD GPUs in Podman is to allow containers to use host devices (container_use_devices is off by default):

sudo setsebool -P container_use_devices 1

The bootstrap assumes the upstream /opt/rocm layout for the HIP compiler

Once hipconfig is on PATH the HIP branch fires, and configure then dies because the bootstrap derives the compiler as HIPCXX="$(hipconfig -l)/clang" (cookbook_helpers.py:989). That only holds for AMD's /opt/rocm layout; on Debian, hipconfig -l returns /usr/bin but the amdgpu clang actually lives at /usr/lib/llvm-17/bin/clang, so configure fails with:

-- Adding CPU backend variant ggml-cpu: -march=native
CMake Error at /usr/share/cmake-3.31/Modules/CMakeDetermineHIPCompiler.cmake:49
(message):
  Could not find compiler set in environment variable HIPCXX:

  /usr/bin/clang.

Call Stack (most recent call first):
  ggml/src/ggml-hip/CMakeLists.txt:43 (enable_language)


CMake Error: CMAKE_HIP_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!

Solution(s)

export HIPCXX=/usr/lib/llvm-17/bin/clang
# or
ln -s /usr/lib/llvm-17/bin/clang /usr/bin/clang

With that, the whole toolchain (hipcc, hipblas, rocblas, hsa-rocr, comgr, device libs at /usr/lib/llvm-17/lib/clang/17/amdgcn/bitcode) resolves cleanly and configure proceeds to the version gate below.

Current llama.cpp requires HIP version 6.1 which isn't in repos

As of writing this, Debian Trixie only has HIP 5.7 (5.7.1). Even with the HIPCXX fix, configure aborts at ggml/src/ggml-hip/CMakeLists.txt:54-55:

if (${hip_VERSION} VERSION_LESS 6.1)
    message(FATAL_ERROR "At least ROCM/HIP V6.1 is required")

Solution(s)

A: pin llama.cpp to a version that requires HIP 5.7

B: Install official AMD ROCm ≥ 6.1 instead of the Debian package. The upstream /opt/rocm layout then satisfies the previous blockers at once: /opt/rocm/bin/hipconfig on PATH trips the cookbook_helpers.py:986 gate, and $(hipconfig -l)/clang resolves to the real ROCm clang — so the HIPCXX/ln workaround above becomes unnecessary.

AMD's apt repo (7.2.4; AMD's Debian 13 instructions map to the Ubuntu noble repo) can be registered inside the container with the steps from AMD's Debian native installation guide:

apt-get update
apt-get install -y wget gnupg ca-certificates

mkdir -p /etc/apt/keyrings
wget -qO - https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor > /etc/apt/keyrings/rocm.gpg
chmod 0644 /etc/apt/keyrings/rocm.gpg

cat > /etc/apt/sources.list.d/rocm.list <<'EOF'
deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/7.2.4 noble main
EOF

cat > /etc/apt/preferences.d/rocm-pin-600 <<'EOF'
Package: *
Pin: release o=repo.radeon.com
Pin-Priority: 600
EOF

apt-get update

apt-get install --no-install-recommends rocm-hip-runtime-dev hipblas-dev rocblas-dev # +8178 MB

With this, llama.cpp rebuilds with ROCm support and serves a working model, resolving both of the earlier build problems (no ROCm userspace in the image, and the HIPCXX /opt/rocm layout assumption).

Unfortunately this is a large install (~8 GB). ROCm distributes prebuilt GEMM/solver kernels, and the bulk is two hard dependencies that upstream's find_package(hipblas/rocblas REQUIRED) makes load-bearing: hipblaslt (~4.8 GB, depended on by rocblas) and rocsolver (~1.7 GB, depended on by hipblas).
Adding this in would require a refactor of the Dockerfiles/composefiles, eg. adding an Odysseus-ROCM image variant or something like that in order to not make the main image heavier than needed.

Vulkan

llama.cpp is unable to build with Vulkan support

The build starts, but cmake configure fails due to missing glslc, SPIRV-Headers, libvulkan-dev.

Solution(s)

I was able to build llama.cpp for Vulkan with the following:

apt-get install libvulkan-dev glslc spirv-headers # +44.6 MB

Build log seen later:

[odysseus] No matching prebuilt llama-server for this host (arch=x86_64) — will build from source.
[odysseus] Vulkan-capable GPU detected (no ROCm/CUDA toolchain installed) — building llama-server with Vulkan support...

llama.cpp is unable to serve models using Vulkan

After fixing the Vulkan build above:

warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions

This is caused by missing Vulkan ICD.

Solution(s)

apt-get install mesa-vulkan-drivers # +83.1 MB

At this point I'm able to serve a GGUF model with llama.cpp using the Vulkan build.

---

The above fixes are only useful to users with AMD GPUs, because NVIDIA ships their own Vulkan ICD with the driver/toolkit.
Splitting the Odysseus container into different versions with one of them being AMD is probably an unavoidable path here.
AMD has their own equivalent of CUDA toolkit container, but it would have to be used by basing the odysseus image on it. This is not feasible, as that container is even bigger than the minimal package set required here.

Alternatively the packages could be put in as a dependency install in the Cookbook, though that can get tedious quick due to no persistence (system packages, not pip packages in /app/data/local) on container recreation. Native llama.cpp build works the same way so a possible consideration. Side note that llama.cpp build with ROCM support is a whole lot heavier/longer than other builds I've seen.

Universal

The build result/return code is ignored - the runner sees no binary and instead uses the default python shim that is a CPU-only wheel.

AI summary of this problem:

• The Linux accel build block chains cmake -B … && cmake --build … && ln -sf … with && (end of _append_llama_cpp_linux_accel_build_lines, cookbook_helpers.py:1030). A configure failure leaves no binary but the script keeps going — no set -e, no exit-code check, no message.
• The next gate, cookbook_routes.py:2249, interprets "no llama-server + import llama_cpp works" as the designed fallback → writes the Python shim (L2251-2282) which execs python3 -m llama_cpp.server. CPU-only.
• Result: Cookbook "succeeds" via its own graceful-degradation path, with the real error only visible to the web GUI by seeing that the model is in fact running on CPU.

Solution(s)

Not ignoring the build failures silently.

Issue breakdown

1. fix(build): install ROCm from AMD repo instead of Debian packages — covers no-ROCm + HIPCXX layout + version gate
2. docs(amd): note SELinux container_use_devices requirement for AMD GPUs
3. fix(build): add Vulkan build dependencies to bootstrap — libvulkan-dev, glslc, spirv-headers, mesa-vulkan-drivers
4. fix(build): check llama.cpp build exit code instead of silently falling back to CPU
5. (optional) fix(build): install mesa-vulkan-drivers for Vulkan runtime ICD — or fold into point 3

This post is up to date as of commit 3b6c169.
This post was written half-manually half-assisted by AI.

[AylxHy]

As a Halo Strix user, I'd love to see this implemented. Many headaches and sanity checks have been made trying to do this myself a s a non-coder. I love this project and really would also love to see full support for AMD!

[shadow-absorber]

yeah this is more or less a must have feature... considering how great the 7900XTX is for running local models with ROCm it is a shame it is half baked when using this frontend