Bitsandbytes dos not support ROCm

[luanrivello]

While trying to run nix run .#textgen-amd I got the following error:

error:
… while evaluating the attribute 'optionalValue.value'

     at /nix/store/xjviahzwa7x51vl51kc3c1k1n1jmhpd5-source/lib/modules.nix:854:5:

      853|
      854|     optionalValue =
         |     ^
      855|       if isDefined then { value = mergedValue; }

   … while evaluating a branch condition

     at /nix/store/xjviahzwa7x51vl51kc3c1k1n1jmhpd5-source/lib/modules.nix:855:7:

      854|     optionalValue =
      855|       if isDefined then { value = mergedValue; }
         |       ^
      856|       else {};

   (stack trace truncated; use '--show-trace' to show the full trace)

   error:

   text-generation-webui is not supported on AMD yet, as bitsandbytes does not support ROCm.

Also may I ask why not keep both KoboldAI and textgen as a part of the project?

[Airradda]

ROCm is not officially supported by bitsandbytes, see #107 #681, so for now, it's set to throw an error when you try to use nix run .#textgen-amd. However, there is PR #756 for adding support. I have managed to get the webui up and running just fine by swapping with the version in the PR, but run into missing module errors when trying to use any the model loaders. I am unsure if this is an issue related to swapping bitsandbytes or something else entirely. If, however, I get it working I will let you know and maybe make a PR if Nixified is okay with using an update to date fork.

As for KobaldAI, while the main repo has been fairly inactive, Henk717 has a pretty active fork with a few more (2827) commits that could be viable to switch to. Alternatively KobaldCPP and KobaldCPP-ROCm could be packaged, although two separate repos seems less than ideal.

[MatthewCroughan]

@Airradda yes, totally up for using a fork, maybe Nixpkgs is also okay with it, but I'll accept it as soon as you make it. For me, this repo is about making things work, regardless of hacks, and then eventually upstreaming as much as possible to Nixpkgs.

[devloic]

text-generation webui works with rocm but installation can make you sweat.

You indeed need to you use bitsandbytes from this repo but you ' ll have to run
ROCM_HOME="/opt/rocm" make hip ROCM_TARGET=gfx1030 (replace gfx1030 with your amd GPU arch)
inside bitsandbytes-rocm-5.6 before any "pip install ." .

textgeneration webui can use many different model loaders ( AutoGPTQ, exllama, llama_cpp ...) and many of those python loaders are rocm specific.

I'll paste a some Dockerfiles I created to run textgen and a requirements.txt for rocm below that worked for me. Sorry I have exactly a 4 hour experience with nix so I can´t translate that to a flake for now.

I don't know if nixified-ai pulls rocm 5.7 like python310Packages.torchWithRocm does but I would stick to rocm 5.6.1 for now since 5.7 is really new and I am not sure that all the python rocm packages work with 5.7 .

Since I have created many other docker containers for amd rocm with apps like : fooocus, h2ogpt, llava, stable-diffusion, Retrieval-based-Voice-Conversion-WebUI I'd be happy to help including them to nixified-ai . Actually docker is what brought me to nix with the hope nix could reduce their size since most of them use around 20GB.

Dockerfile rocmyai-base:

FROM rocm/dev-ubuntu-22.04:5.6.1-complete as rocmyai-base

SHELL ["/bin/bash", "-c"]
RUN apt-get update
&& apt-get install wget nano sudo git gcc build-essential -y && useradd -rm -d /home/ai -s /bin/bash -g root -G sudo -u 1001 ai -p "$(openssl passwd -1 ai)" && usermod -aG render ai
WORKDIR /home/ai
USER ai
ENV PATH="/home/ai/miniconda3/bin:$PATH"
RUN mkdir -p ~/miniconda3 && wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
&& chmod +x ~/miniconda3/miniconda.sh
&& ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
&& rm -rf ~/miniconda3/miniconda.sh
&& conda install python=3.10
&& conda init bash
&& conda create --name rocmyaienv
&& echo "PATH="/home/ai/miniconda3/bin:$PATH"" >> ~/.bashrc
SHELL ["conda", "run", "-n", "rocmyaienv", "/bin/bash", "-c"]
RUN pip install https://download.pytorch.org/whl/nightly/pytorch_triton_rocm-2.1.0%2B34f8189eae-cp310-cp310-linux_x86_64.whl
&& pip install https://download.pytorch.org/whl/nightly/rocm5.6/torch-2.2.0.dev20230921%2Brocm5.6-cp310-cp310-linux_x86_64.whl
&& pip install https://download.pytorch.org/whl/nightly/rocm5.6/torchvision-0.17.0.dev20230921%2Brocm5.6-cp310-cp310-linux_x86_64.whl
&& pip cache purge

Dockerfile textgenwebui_rocmyai-5.6.1:

FROM rocmyai-base as textgenwebui_rocmyai-5.6.1

SHELL ["/bin/bash", "-c"]
WORKDIR /home/ai
USER ai
ADD requirements_rocm.txt ./
SHELL ["conda", "run", "-n", "base", "/bin/bash", "-c"]
RUN conda rename -n rocmyaienv textgenenv
SHELL ["conda", "run", "-n", "textgenenv", "/bin/bash", "-c"]
RUN git clone https://github.com/arlo-phoenix/bitsandbytes-rocm-5.6 && cd bitsandbytes-rocm-5.6 && git checkout e38b9e91b718e8b84f4678c423f72dd4decce4e5
&& ROCM_HOME="/opt/rocm" make hip ROCM_TARGET=gfx1030 && pip install . && cd ..
&& git clone https://github.com/devloic/AutoGPTQ && cd AutoGPTQ && git checkout fadb2bafce5a63226312cef7d5c807c907b86822
&& ROCM_VERSION=5.6.1 pip install -e . && cd ..
&& git clone https://github.com/jllllll/exllama && cd exllama && git checkout 86cd5c00123b4761003aac8224114ba35755c54b
&& ROCM_VERSION=5.6.1 python3 setup.py install && cd ..
&& pip install -r requirements_rocm.txt
&& git clone https://github.com/oobabooga/text-generation-webui.git && cd text-generation-webui && git checkout f63361568c476ff4b2414a06027bc22e6123941f
&& echo "conda activate textgenenv" >> ~/.bashrc
&& pip cache purge
WORKDIR /home/ai/text-generation-webui

requirements_rocm.txt:

accelerate==0.23.*
colorama
datasets
einops
exllamav2==0.0.5
gradio==3.47.*
markdown
numpy==1.24
optimum==1.13.1
pandas
peft==0.5.*
Pillow>=9.5.0
pyyaml
requests
safetensors==0.4.0
scipy
sentencepiece
tensorboard
transformers==4.34.*
tqdm
wandb
git+https://github.com/oobabooga/torch-grammar.git
llama_cpp_python @ https://github.com/jllllll/llama-cpp-python-cuBLAS-wheels/releases/download/rocm/llama_cpp_python-0.2.9+rocm5.6.1-cp310-cp310-manylinux_2_31_x86_64.whl#sha256=a52841046284c3664c12c8cad85d93e6bf08b4b6aaf9db313e86a601f127
5c6f
https://github.com/jllllll/GPTQ-for-LLaMa-CUDA/releases/download/0.1.1/gptq_for_llama-0.1.1+rocm5.6-cp310-cp310-linux_x86_64.whl

[Airradda]

@Airradda yes, totally up for using a fork, maybe Nixpkgs is also okay with it, but I'll accept it as soon as you make it. For me, this repo is about making things work, regardless of hacks, and then eventually upstreaming as much as possible to Nixpkgs.

I'll start working on it after I figure out how the new structure is setup after the refactor (079f83b). My working copy simply overwrote bitsandbytes as I only have an AMD GPU and didn't need the Nvidia stuff, which obviously won't work here.

It would probably be a good idea to have some info for contributing along with any relevant info for #2 and #26.

[MatthewCroughan]

@devloic

since most of them use around 20GB.

The reason they use 20GB is because of the poorly defined and seemingly unlimited runtime dependencies. text-generation-webui ambiguously requires the kitchen sink a runtime, and if we want to make the functionality equivalent to what you just posted, we also have to provide the kitchen sink. We also have to reverse engineer what parts the kitchen sink is made of, because not all of the dependencies are cleanly described in their entirety in any readable file or document in the repo, such as the requirements.txt.

If we obtained equivalent functionality, it's probably true that we'd approach that size too. I don't think text-generation-webui is worth that amount of effort though, as something written in Rust compatible with WGPU/BURN is hopefully going to be written in the future, which means all of these problems will disappear, as it's created on better foundations.

I will accept any PR that fixes text-generation-webui's quirks though.

[Airradda]

For me the instant gain is not having to have 2+ copies of ROCm and some of the other requirements as I have Text-Gen (docker), SD-WebUI(nix-stable-diffusion), and InvokeAI(nixified-ai) installed, having multiple copies of models as not everything is playing nice with symlinks (docker), and basically just not having to deal with docker.

Tackling extensions and their requirements (the kitchen sink) is going to be a whole nother mess.