ROCm support

[mjsulik]

It looks like CUDA can be much faster than --device cpu, especially as model size increases.
For example, https://www.assemblyai.com/blog/how-to-run-openais-whisper-speech-recognition-model/#whisper-advanced-usage

At least one person has appeared to have success using an AMD consumer GPU (RX 6800). https://news.ycombinator.com/item?id=32933236

Options for --device appear to include: cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, ort, mps, xla, lazy, vulkan, meta, hpu, privateuseone

I have the appropriate version of torch installed on Linux, but I'm not sure how to enable ROCm using --device

[jongwook]

We haven't tested ROCm, but from this documentation it seems that you can keep using cuda if the ROCm version is properly installed.

PyTorch for HIP intentionally reuses the existing torch.cuda interfaces. This helps to accelerate the porting of existing PyTorch code and models because very few code changes are necessary, if any.

[mjsulik]

Thank you, using "--device cuda" was successful after correctly configuring ROCm/HIP. In case it helps anyone else, I needed to install rocm-libs and set environment variable HSA_OVERRIDE_GFX_VERSION=10.3.0 to make this work.

[DrMemoryFish]

how would i do this for gforce rtx 2060?

[marcin-przywoski]

@mjsulik can you tell me how much of a performance gain it was? Were you using WSL2?

[mjsulik]

I was using Linux, not WSL2.

I actually didn't finish transcription for a file using the CPU because it was so slow on the medium.en model. I can't quantify the performance difference, but it was definitely large! If your only option is to use the CPU, then I think you'd want to start with a small model size (tiny or base) to find out how long it takes and then you could evaluate if the quality is acceptable. The main Github page for this project (https://github.com/openai/whisper) seems to assume you're using a GPU (e.g., it references the VRAM needed for different models), but I'd assume that the time scaling across models (e.g., x32 for tiny, x16 for base) would be consistent whether you're using CPU or GPU.

[EchedeyLR]

Thank you, using "--device cuda" was successful after correctly configuring ROCm/HIP. In case it helps anyone else, I needed to install rocm-libs and set environment variable HSA_OVERRIDE_GFX_VERSION=10.3.0 to make this work.

Could you attach more instructions @mjsulik ? I am unsure if you imply all the libraries following the AMD documentation, the ROCm usecase (guessing is being installed as that) or the HIP usecase.

[BranML]

If it supports ROCm, it will be great.

[chowder3907]

You can already use pytorch-rocm to take advantage of AMD GPUs. Install it through pip before installing openai

[GrahamboJangles]

What's the secret to installing pytorch-rocm, because when I try it says it doesn't exist.

[NielsMayer]

ROCm works fine with Whisper. I'm using it with "Radeon RX Vega (VEGA10, DRM 3.42.0, 5.15.0-48-generic, LLVM 12.0.0)".
With this GPU, I can process media using up to the 'medium' model. The large model is too big for this old 8Gb RxVega 56 Card.

A few important steps in getting it working (i'm on Kubuntu 20.04.5 w/ kernel 5.15.0-56-generic , YMMV):

(1) sudo apt install clinfo mesa-opencl-icd
(2) add your username to group 'render' in addition to 'video' to allow access to /dev/dri/* (and logout/login on desktop, or login to a new session if on server to propagate these important permissions)
(3) see if your card shows up in 'clinfo'
(4) sudo pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/rocm5.2
(5) in python3 see if torch sees the GPU and CUDA:

>>> import torch
>>> torch.cuda.is_available() 
True
>>> torch.cuda.device_count()
1
>>> torch.cuda.get_device_name(0)
'Radeon RX Vega'

(6) Now install whisper and check "whisper --help" to see if it outputs:
--device DEVICE device to use for PyTorch inference (default: cuda)

With my GPU, whisper outputs the following warning:

MIOpen(HIP): Warning [SQLiteBase] Missing system database file: gfx900_56.kdb Performance may degrade. Please follow instructions to install:
https://github.com/ROCmSoftwarePlatform/MIOpen#installing-miopen-kernels-package

However I've seen suggestions that the above warning is spurious, and in either case, applies to the AMD proprietary drivers whereas I'm using the open-source default 'amdgpu' drivers.

Anybody have further info on resolving this warning for open source 'amdgpu' users?? Will I see better performance using AMD's proprietary drivers??

........

On a different system, I had the hare-brained idea of trying to use my AMD 4750G "APU" (Ryzen with integrated AMD GPU) with pytorch. In this case I ended up using the "nightly" ROCm build versions of "torch 2.0.0.dev20221219+rocm5" and torchaudio and torchvision suggested by https://pytorch.org/ .

After much failure, trying a hint from https://stackoverflow.com/questions/73229163/amd-rocm-with-pytorch-on-navi10-rx-5700-rx-5700-xt ,
I set environment variable HSA_OVERRIDE_GFX_VERSION=10.3.0 . With this, at least at the pytorch level, it claims CUDA works on this APU:

$ HSA_OVERRIDE_GFX_VERSION=10.3.0 python3
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.cuda.device_count()
1
>>> torch.cuda.get_device_name(0)
'AMD Radeon Graphics'

Note that without the env-var, whisper pukes:

$ whisper --help
"hipErrorNoBinaryForGpu: Unable to find code object for all current devices!"
Aborted (core dumped)

With the env-var, whisper claims cuda works:

$ HSA_OVERRIDE_GFX_VERSION=10.3.0 whisper --help
...
--device DEVICE       device to use for PyTorch inference (default: cuda)
...

However running a transcription with most models runs out of memory and dumps core (bug that it dumps core??):

Memory access fault by GPU node-1 (Agent handle: 0x8a46980) on address (nil). Reason: Page not present or supervisor privilege.
Aborted (core dumped)

With the 'base' or 'tiny' models, it no longer dumps core, but instead hangs forever at 100% CPU, outputting nothing.

Furthermore, in this 'hung' state, whisper no longer can be ^C'd to kill the hanging process (bug?). Rather the process needs to be killed externally, or you need to background the process and then "kill %1" it.

Note that radeontop(1) (from https://github.com/clbr/radeontop not apt) indicates how little memory is available on AMD 4750G with a running KDE desktop:

472M / 391M VRAM 120.68% │
1837M / 3063M GTT  59.98%

(NB: i've also tried this AMD4750G APU system "headless" so that GPU is only used for whisper and not to run the KDE desktop, and it still hangs or crashes as above).

And with that, I've given up on the AMD 4750G APU and continue to use my old Power-hog RxVega 56 (165W) with the 'medium' model quite successfully: https://www.youtube.com/watch?v=AFk5g7NJ1Ko https://rumble.com/v1n7cx8-trainspodder-and-whisper-transcribes-radio-w-good-proper-noun-spelling-infe.html

In contrast, for the RxVega56 radeontop(1) reports (on a "headless" system where I'm using the GPU 100% for whisper).

49M / 8119M VRAM   0.60%
11M / 8164M GTT   0.13%

[colemannugent]

Got it working on a 5700XT using these instructions. The "medium" model uses ~97% of my 8 GB of VRAM.

I had to add --fp16 False to my whisper invocation otherwise I got nonsense results from the transcription.

I'm on Ubuntu 22.04.2 LTS with kernel 5.15.0-60-generic.

[jp-berg]

Adding for less technical people like me: Order of installation is important. Install all components in a fresh virtual enviroment! And ROCM 5.45 is currently not working for me (with Vega56), you will need 5.2, e.g. pip install torch==1.13.0+rocm5.2 torchvision==0.14.0+rocm5.2 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/rocm5.2

[Whitesttax]

Adding for less technical people like me: Order of installation is important. Install all components in a fresh virtual enviroment! And ROCM 5.45 is currently not working for me (with Vega56), you will need 5.2, e.g. pip install torch==1.13.0+rocm5.2 torchvision==0.14.0+rocm5.2 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/rocm5.2

I was trying to install on wsl2 so I figured that's why it's not working. I have very little experience with linux, if I decide to install it is that really all the lines I have to run to get it working?

[gnevan]

Suppose you should be able to allocate more memory for your APU in BIOS, I think up to 2GB (depending on your motherboard).

[albcunha]

Hijacking this thread, I had a hard time get things to work on docker.
We have a RX6700XT. On medium, it transcribes 8x normal speed, or more. We also implemented SILERO VAD, so that may cut some time too.
Here is a docker file and docker compose that I use for rocm on a AMD RX6700XT:
#docker-compose.yml

version: "3.8"
services:
  worker:
      restart: always
      container_name: transcricao-redis-worker
      image: gaeco/masterimage:1.0
      # stdin_open: true # docker run -i
      # tty: true        # docker run -t
      # entrypoint: /bin/sh
      depends_on:
        - redis
      command: python3 worker.py
      shm_size: 8gb
      group_add:
        - video
        - render
      cap_add:
         - SYS_PTRACE 
         
      security_opt:
        - seccomp=unconfined
      
      devices:
        - /dev/kfd
        - /dev/dri

  
#dockerfile
FROM rocm/dev-ubuntu-20.04:5.4-complete@sha256:fa56caaeb6b6a61a04e679cfb9ebb585cd2634585b03b01d867c5df7025840b3

# FROM rocm/pytorch-nightly:2023-01-22-rocm5.4
ENV HSA_OVERRIDE_GFX_VERSION=10.3.0
ENV LD_LIBRARY_PATH=/opt/rocm/lib
ARG DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED=1
ENV APP_HOME=/home/user
ENV PATH=$PATH:/home/user/.local/bin:/opt/rocm-5.4/lib:/opt/rocm-5.4/bin:/opt/rocm-5.4.2/opencl/bin
RUN apt-get update
RUN apt-get install -y --fix-missing ffmpeg libjpeg-dev python3-dev python3-pip git

RUN mkdir –m777 $APP_HOME
WORKDIR $APP_HOME


RUN mkdir -p –m777 $APP_HOME/logs 
RUN mkdir -p –m777 $APP_HOME/.cache/whisper
RUN mkdir -p –m777 $APP_HOME/.cache/torch/hub

RUN adduser --disabled-password --gecos '' --shell /bin/bash user
# RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-user
RUN usermod -a -G render,video user
RUN chown -R user:user $APP_HOME/

USER user

WORKDIR $APP_HOME

COPY --chown=user:user . $APP_HOME/

RUN pip install wheel setuptools
RUN pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/rocm5.2
RUN pip install -r requirements.txt 

USER root
RUN sudo ln -s /opt/rocm-5.4.0/lib/libroctx64.so /opt/rocm-5.4.0/lib/libroctx64.so.1
RUN chown -R user:user $APP_HOME/.cache/whisper
RUN chown -R user:user $APP_HOME/.cache/torch/hub
USER user

Please, consider that I know nothing, so there my be errors or unecessary steps on these files, but I took a long time to figure it out.

You also need to install rocm drivers on your host machine first. First I suggest to get: torch.is_cuda_available() to True and doing some testes locally before going to docker.

[Whitesttax]

Thank you for the explanation. I'm trying to follow it exactly as you wrote, this line gives me trouble:

wget -qO - http://repo.radeon.com/rocm/apt/debian/rocm.gpg.key | sudo apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
gpg: no valid OpenPGP data found.

Tried: Remove the Certification Check, Install certificates, Change the Path in the .bashrc File
"http://repo.radeon.com/rocm/apt/debian/rocm.gpg.key" gives 404 error on browser, maybe that's why?

[mjsulik]

I'm not able to find that any longer in the documentation, so maybe it is no longer needed. Are you able to run the following commands successfully without the key?

That comes from this guide, and I'm not sure if that key is really necessary. https://sep5.readthedocs.io/en/latest/Installation_Guide/Installation-Guide.html

The AMD installation guide differs at this point:

Ubuntu v20.04

echo 'deb [arch=amd64 signed-by=/etc/apt/trusted.gpg.d/rocm-keyring.gpg] https://repo.radeon.com/rocm/apt/5.4.3 focal main' | sudo tee /etc/apt/sources.list.d/rocm.list
echo -e 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' | sudo tee /etc/apt/preferences.d/rocm-pin-600
sudo apt-get update

Ubuntu v22.04

echo 'deb [arch=amd64 signed-by=/etc/apt/trusted.gpg.d/rocm-keyring.gpg] https://repo.radeon.com/rocm/apt/5.4.3 jammy main' | sudo tee /etc/apt/sources.list.d/rocm.list
echo -e 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' | sudo tee /etc/apt/preferences.d/rocm-pin-600
sudo apt-get update

[RaymondTheBrave0]

Can anyone tell me what is the bests setup to try and get Roc support for AMD Radeon X570 GPU to work for Whisper.

I have Linux Mint 21.2 at the moment running on a Aorus Ultra X570 Mobo with 32GB Ram and with a Ryzen 5 Cpu. I have tried so many things to try and get Whisper to use the GPU (8GB GDDR5 Ram) and have now decided that using a VM on Virtualbox would be a good way to see if I can get it to work. I was looking at using Ubuntu as my base operating system. Is this a good choice and will Virtualbox allow pytorch and Roc functionality to see the GPU properly so that it will use it.

If someone could suggest the steps to take in order to get a successful run of whisper using my AMD GPU on my system.

I have started now using anaconda which seems to help keep things in some sort of order and knowing what has been installed.

If I issue the clinfo command I get the following:-

clinfo
Number of platforms 1
Platform Name Clover
Platform Vendor Mesa
Platform Version OpenCL 1.1 Mesa 23.0.4-0ubuntu1~22.04.1
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_icd
Platform Extensions function suffix MESA

Platform Name Clover
Number of devices 1
Device Name AMD Radeon RX 570 Series (polaris10, LLVM 15.0.7, DRM 3.42, 5.15.0-88-generic)
Device Vendor AMD
Device Vendor ID 0x1002
Device Version OpenCL 1.1 Mesa 23.0.4-0ubuntu122.04.1
Device Numeric Version 0x401000 (1.1.0)
Driver Version 23.0.4-0ubuntu122.04.1
Device OpenCL C Version OpenCL C 1.1
Device Type GPU
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Max compute units 32
Max clock frequency 1284MHz
Max work item dimensions 3
Max work item sizes 256x256x256
Max work group size 256
Preferred work group size multiple (kernel) 64

Which looks promising and if gointo python I get the following responses:-

python3
Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux

import torch
torch.cuda.is_available()
True
torch.cuda.device_count()
1
torch.cuda.get_device_name(0)
'AMD Radeon RX 570 Series'

However when I issued this command I got a blank, echo $CUDA_VISIBLE_DEVICES
Should it not be set to at least 1.

I have been able to run whisper but it does not use my AMD Gpu based on Mission Center graph of GPU usage, which I hope I can believe. It was also runs very slowly and the CPU usage is up at 60%.

[RaymondTheBrave0]

I have realised that in trying to use a Virtualbox environment and I am going to have passthru issues as the Ubuntu cannot see the hardware directly so I would assume I cant test whisper using the AMD GPU. So I will not going ahead with a VM test. I will look at maybe setting up a USB Live bootable environment to see if I can get whisper using my AMD GPU.

[webdevjosue]

I have realised that in trying to use a Virtualbox environment and I am going to have passthru issues as the Ubuntu cannot see the hardware directly so I would assume I cant test whisper using the AMD GPU. So I will not going ahead with a VM test. I will look at maybe setting up a USB Live bootable environment to see if I can get whisper using my AMD GPU. amd rocm gpu support rdna3
You may have better chance on 7900xt/xtx from what I gather it's cpu only without cuda cores.

[viebrix]

Here is a short description how to use Whisper with older AMD Cards (GFX803) RX 580
https://github.com/viebrix/pytorch-gfx803-for-Whisper

Yesterday I managed to get Whisper (or Whisper-WEBUI) start and running with GPU (GFX803) RX 580 (8GB).