ROCm support for AMD GPUs on Linux

[WorXeN]

• Added requirements for AMD GPUs
• Updated README with instructions on how to use RVC with ROCm

Pull request checklist

• The PR has a proper title. Use Semantic Commit Messages. (No more branch-name title please)
• Make sure you are requesting the right branch.
• Make sure this is ready to be merged into the relevant branch. Please don't create a PR and let it hang for a few days.
• Ensure all tests are passing.
• Ensure linting is passing.

PR type

• new feature

Description

• Adds descriptions and requirements, on how to use ROCm for AMD GPUs on Linux

Screenshot

Localhost url to test on

Jira Link

Ticket

[NatoBoram]

Is there a specific version of the AMD drivers to use? Oftentimes, ML packages like to break across minor version and environments. If you have tested it to work on a specific version, can you provide it?

[WorXeN]

I'm using the latest mesa 23.1.6-3 on arch. For ROCm i'm running version 5.6.0-1 and i've tested pytorch with ROCm 5.6 aswell as with ROCm 5.4.2

[fumiama]

It is always welcomed to provide a detailed document.

[NatoBoram]

Worth mentioning that the step "2. Add the AMDGPU Repository and Install the Kernel-mode Driver" can be skipped on most Linux distributions since they'll include the AMD GPU driver. Not only that, but doing this can actually harm your current installation.

[Chilluminati91]

AMD GPU is recognized at startup now, but "train" tab still shows "no supported GPU". All GPU ID fields are empty, and if I manually set them to "0" the GPU still does not get utilized, neither in pitch extraction nor in training (which immediately errors out)

/home/liam/Retrieval-based-Voice-Conversion-WebUI/venv/lib/python3.10/site-packages/torch/cuda/__init__.py:546: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
2023-09-21 10:41:48.122299: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-09-21 10:41:48.155038: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-21 10:41:49 | INFO | faiss.loader | Loading faiss with AVX2 support.
2023-09-21 10:41:49 | INFO | faiss.loader | Successfully loaded faiss with AVX2 support.
2023-09-21 10:41:49 | INFO | configs.config | Found GPU AMD Radeon RX 6650 XT
2023-09-21 10:41:49 | INFO | __main__ | Use Language: en_US
Running on local URL:  http://0.0.0.0:7865

[JohnAMacDonald]

Posted in AMD Segmentation Fault as well as here for visibility.

So, after much trial and error, I got this working, I will list the instructions below on how I got Training working on an AMD RX 6750 XT on Ubuntu Desktop 22.04.3 LTS as per lsb_release -a.

Some notes:

I downloaded the code directly, not a release version, it should work on a release version, but I downloaded the code, ran through the build instructions etc.

A big thanks to this person on Reddit who wrote the original base instructions I used and tweaked for RVC Orion_light on Reddit

ALSO, I haven't tested rmvpe or rmvpe_gpu as I forgot to get the pretrain's for them, but it should work, side note I believe rmvpe was having issues with audio longer than 3 minutes, at least I was.

Install Notes:

1. Download and install Ubuntu Desktop 22.04.3 LTS from the official Ubuntu website.
2. Once installed open a terminal window.
3. Run these commands separately sudo usermod -a -G render YourUsernameHere & sudo usermod -a -G video YourUsernameHere. This is adding yourself to both the Render & Video Groups..
4. Install Python3 with this command (may already be preinstalled) sudo apt-get install python3.
5. Open Bashrc with Nano with this command nano ~/.bashrc, then at the bottom of that file add alias python=python3 export HSA_OVERRIDE_GFX_VERSION=10.3.0. Make sure that the alias python=python3 & export HSA_OVERRIDE_GFX_VERSION=10.3.0 are on DIFFERENT lines.
6. Reboot (Important).

After booting back into Ubuntu, we will install ROCm and Pytorch.

7. Go to the PyTorch | Start Locally website and check the ROCm version (currently 5.4.2).
8. Go to this website How to Install ROCm (amd.com). select version that is compatible with pytorch and find command for you installed ubuntu version. Example that works right now with RVC V2 sudo apt-get update wget https://repo.radeon.com/amdgpu-install/5.4.2/ubuntu/jammy/amdgpu-install_5.4.50402-1_all.deb sudo apt-get install ./amdgpu-install_5.4.50402-1_all.deb.
9. Then run this command sudo amdgpu-install --usecase=rocm --no-dkms and let that finish.
10. Reboot (Important).
11. Open Start Locally | PyTorch again, select Stable, Linux, pip, python, and ROCm. and run the command outputted below that in your terminal. Example pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.2. (You may need to install pip, with Ubuntu it will just let you know that pip is missing and you can get it by running something like sudo apt-get install pip. That pip install command could be wrong, so double check.
12. Reboot (Important).

Next we will build RVC V2 from source, pretty self explanatory via the official docs, but will retype them here as there is some extra stuff with AMD on Linux

13. Download the source code either by clicking Code then Download ZIP or git clone https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI.git
14. Extract the ZIP file.
15. CD (Change Directory) into the the extracted folder.
16. Run this command curl -sSL https://install.python-poetry.org | python3 -.
17. After that Finishes, run this command poetry install.
18. After that, then install the projects AMD requirements via pip with this command pip install -r requirements-amd.txt.
19. Then after that is done, run this command sudo apt-get install rocm-hip-sdk rocm-opencl-sdk.
20. Don't know if this does anything important, but doing these steps made mine work, so run the command export ROCM_PATH=/opt/rocm but not the command export HSA_OVERRIDE_GFX_VERSION=10.3.0 as we already added that to our Bashrc file.
21. After everything is done, run the command python3 infer-web.py to open the web interface to start using RVC!

Note about the interface: I had to use Harvest like instead of rmvpe or rmvpe_gpu becuase I forgot to download that model. Also for GPU indexes I put 0-1-2 just to be safe. And the biggest note your GPU WON'T show under GPU Information, it will say Unfortunately, there is no compatible GPU available to support your training., but when you go to train, open a new terminal window and run the command rocm-smi and it will tell you this info from left to right: GPU Index (I believe) Temp AvgPwr SCLK MCLK Fan Percentage Perf PwrCap VRAM% and GPU%. To tell if your AMD card is being used, check the Temp and the GPU%.

At a batch size of 16 and training 300 Epochs I'm using 99% of my GPU as indicated by GPU% and my temperature is around the low to mid 70s in celsius, I also do have some Coil Whine (Reference AMD GPU). It also takes about 30-40 seconds per Epoch.

I hope this helped someone trying to set this up and train with their AMD GPU on Linux!

[Skynetaa]

is it possible to run RVC on AMD in Windows ? I have RX6800 XT but console says no supported Nvidia GPU found