Skip to content

feat(ROCm): Allow selecting ROCm version when building llama.cpp backend #7615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
sredman wants to merge 12 commits into
base: master
Choose a base branch
from
+43 −4
Draft

feat(ROCm): Allow selecting ROCm version when building llama.cpp backend #7615

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
539b77c
Add build arg for ROCm version
sredman Sep 23, 2025
05f0247
Break ROCM_VERSION into ROCM_{MAJOR,MINOR}_VERSION
sredman Sep 24, 2025
2a63e2e
Use correct ROCm package names
sredman Sep 24, 2025
fbee741
Add rocm package for runtime libs
sredman Sep 25, 2025
df64b1b
Remove hipblas-dev and rocblas-dev. I think they are not needed, and …
sredman Sep 25, 2025
5110dab
Migrate ROCM build flags to llama-cpp docker file
sredman Dec 16, 2025
cccab5b
Change base-level Dockerfile back to same runtime dependency packages…
sredman Dec 18, 2025
c050a01
Use default ROCm version of 5.5.1 to match Ubuntu package
sredman Dec 18, 2025
1c698ce
Remove ROCm runtime package from llamacpp dockerfile
sredman Dec 18, 2025
31e0f5e
Annotate ROCm runtime dependencies
sredman Dec 18, 2025
2c45d57
Remove comments in runtime dependencies install step
sredman Dec 19, 2025
286b2c7
Merge branch 'master' into work/sredman/rocm-version-control
mudler Dec 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ FROM requirements AS requirements-drivers
ARG BUILD_TYPE
ARG CUDA_MAJOR_VERSION=12
ARG CUDA_MINOR_VERSION=0
ARG ROCM_MAJOR_VERSION=5
ARG ROCM_MINOR_VERSION=5.1 # ROCm version to append to the major version, in the format of their apt repo (https://repo.radeon.com/rocm/apt/). Like `0_alpha` or `3.4`.
Copy link
Contributor Author

@sredman sredman Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal is to update the ROCM_MAJOR_VERSION and ROCM_MINOR_VERSION to match whatever is deployed in LocalAI today. I haven't 100% figured this out. I believe it is v5.5.1, as that is the version in the Ubuntu 22.04 repo.

ARG SKIP_DRIVERS=false
ARG TARGETARCH
ARG TARGETVARIANT
Expand Down Expand Up @@ -111,10 +113,26 @@ RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
; fi

RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
# Setup for specific ROCm version as described here: https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/install-methods/package-manager/package-manager-ubuntu.html
ROCM_VERSION="${ROCM_MAJOR_VERSION}.${ROCM_MINOR_VERSION}" && \
apt-get update && \
apt-get install -y --no-install-recommends \
hipblas-dev \
rocblas-dev && \
gpg wget && \
mkdir --parents --mode=0755 /etc/apt/keyrings && \
wget -qO - https://repo.radeon.com/rocm/rocm.gpg.key | gpg --yes --dearmor --output /etc/apt/keyrings/rocm.gpg && \
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/${ROCM_VERSION} jammy main" >> /etc/apt/sources.list.d/rocm.list && \
if [ "${ROCM_MAJOR_VERSION}" -ge 7 ]; then \
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/graphics/${ROCM_VERSION}/ubuntu jammy main" >> /etc/apt/sources.list.d/rocm.list \
; fi && \
echo "Package: *" >> /etc/apt/preferences.d/rocm-pin-600 && \
echo "Pin: release o=repo.radeon.com" >> /etc/apt/preferences.d/rocm-pin-600 && \
echo "Pin-Priority: 600" >> /etc/apt/preferences.d/rocm-pin-600 && \
# End setup steps for specific ROCm version - the packages below will be installed from the configured repositories
apt-get update && \
apt-get install -y --no-install-recommends \
rocm-hip-runtime \
rocblas-dev \
hipblas-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
echo "amd" > /run/localai/capability && \
Expand Down
25 changes: 23 additions & 2 deletions backend/Dockerfile.llama-cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,13 @@ ARG BUILD_TYPE
ENV BUILD_TYPE=${BUILD_TYPE}
ARG CUDA_MAJOR_VERSION
ARG CUDA_MINOR_VERSION
ARG ROCM_MAJOR_VERSION
ARG ROCM_MINOR_VERSION
ARG SKIP_DRIVERS=false
ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
ENV ROCM_MAJOR_VERSION=${ROCM_MAJOR_VERSION}
ENV ROCM_MINOR_VERSION=${ROCM_MINOR_VERSION}
ENV DEBIAN_FRONTEND=noninteractive
ARG TARGETARCH
ARG TARGETVARIANT
Expand Down Expand Up @@ -158,10 +162,27 @@ RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
; fi

RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
# Setup for specific ROCm version as described here: https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/install-methods/package-manager/package-manager-ubuntu.html
ROCM_VERSION="${ROCM_MAJOR_VERSION}.${ROCM_MINOR_VERSION}" && \
apt-get update && \
apt-get install -y --no-install-recommends \
hipblas-dev \
rocblas-dev && \
gpg wget && \
mkdir --parents --mode=0755 /etc/apt/keyrings && \
wget -qO - https://repo.radeon.com/rocm/rocm.gpg.key | gpg --yes --dearmor --output /etc/apt/keyrings/rocm.gpg && \
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/${ROCM_VERSION} jammy main" >> /etc/apt/sources.list.d/rocm.list && \
if [ "${ROCM_MAJOR_VERSION}" -ge 7 ]; then \
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/graphics/${ROCM_VERSION}/ubuntu jammy main" >> /etc/apt/sources.list.d/rocm.list \
; fi && \
echo "Package: *" >> /etc/apt/preferences.d/rocm-pin-600 && \
echo "Pin: release o=repo.radeon.com" >> /etc/apt/preferences.d/rocm-pin-600 && \
echo "Pin-Priority: 600" >> /etc/apt/preferences.d/rocm-pin-600 && \
# End setup steps for specific ROCm version
apt-get update && \
apt-get install -y --no-install-recommends \
# Build dependencies
rocm-developer-tools \
rocm-hip-runtime-dev \
rocm-hip-sdk && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
# I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
Expand Down
Loading