11. ROCm

AMD GPUs (ROCm)

TabbyAPI runs on AMD GPUs through a ROCm build of ExLlamaV3. The dependencies come from AMD's pip-native ROCm distribution: torch and the ROCm runtime install from AMD's package index, so no system-wide ROCm installation is needed, only the amdgpu kernel driver.

Supported hardware

The ExLlamaV3 wheels are built for every RDNA generation in one wheel (RDNA1 through RDNA4, including the Ryzen AI 300 and Strix Halo APUs). RDNA3 discrete cards are what the wheels have been run on; RDNA1 and RDNA2 are compile-tested only so far. Instinct (CDNA) cards are not covered by the wheels; build ExLlamaV3 from source for those. Linux x86_64 with Python 3.12 to 3.14 is supported. On Windows, ExLlamaV3 has to be built from source.

AMD ships torch's device code per chip, so the install needs the GPU's gfx target. The start script reads it from the kernel; to look it up yourself:

grep gfx_target_version /sys/class/kfd/kfd/topology/nodes/*/properties

gfx_target_version	Target	GPUs
110000 / 110001 / 110002	gfx1100 / gfx1101 / gfx1102	Radeon RX 7900, RX 7800 and 7700, RX 7600
120000 / 120001	gfx1200 / gfx1201	Radeon RX 9060, RX 9070
100300 / 100301 / 100302	gfx1030 / gfx1031 / gfx1032	Radeon RX 6800 and 6900, RX 6700, RX 6600
110500 / 110501	gfx1150 / gfx1151	Ryzen AI 300, Strix Halo

Installing

1. With the start script: run start.sh. An AMD GPU is detected from /dev/kfd and its target from the kernel's topology; the targets are saved as rocm_targets in start_options.json. They can be given up front with --gpu-lib rocm --rocm-targets gfx1100, comma-separated for a machine with several different AMD GPUs.

2. Manually, in a virtual environment, naming AMD's two indexes and the device package for each GPU:

   pip install -U .[rocm] \
     --extra-index-url https://stable.repo.amd.com/rocm/pytorch/whl-next/ \
     --extra-index-url https://stable.repo.amd.com/rocm/core/whl-next/ \
     amd-torch-device-gfx1100

TABBY_ROCM_INDEX_URLS (comma-separated) overrides the indexes the start script uses, for a mirror or an offline copy.

The torch build in the rocm extra is AMD's build of torch 2.13.0 for ROCm 10. ExLlamaV3's ROCm wheels are built against exactly that version, and the two are updated together. The indexes also carry newer torch and ROCm lines, which is why everything in the extra is pinned.

Warning

A plain pip install torch replaces the ROCm build with a CPU-only one. TabbyAPI reports this at startup; reinstall with the command above or the update script to recover.

Docker

docker/Dockerfile.rocm builds from plain Ubuntu, with the GPU target(s) as a build argument:

docker build -f docker/Dockerfile.rocm --build-arg ROCM_TARGETS=gfx1100 -t tabbyapi:rocm .

The container needs the GPU devices passed through and membership of the host's video and render groups. Images don't define a render group, so it is given by number. The tabbyapi-rocm service in docker/docker-compose.yml sets this up:

RENDER_GID=$(getent group render | cut -d: -f3) docker compose --profile rocm up

or with plain Docker:

docker run --device /dev/kfd --device /dev/dri --group-add video --group-add $(getent group render | cut -d: -f3) \
  --shm-size 8g -p 5000:5000 -v ./models:/app/models tabbyapi:rocm

Notes

• memory.cuda_malloc_async has no effect on ROCm and is ignored with a warning.
• TabbyAPI sets TRITON_DEFAULT_BACKEND=amd when it finds a ROCm torch, unless already set. AMD's Triton build ships both backends and would otherwise refuse to start on a machine that also has the NVIDIA driver installed.
• The tensor-parallel backends and CPU MoE offload have not been validated on ROCm yet.