LLM on Docker (WSL2)

About

This repository is meant to allow for easy installation of LLM on Windows. One click to install. Second click to start.

It's a direct copy of what https://github.com/rgryta/Stable-Diffusion-WSL2-Docker is for AUTOMATIC1111 webui, but for LLM and using oobabooga webui for LLMs.

Prerequisites

Before following through with these instructions. Please verify below.

1. You have virtualization support - easiest way is to check if you can see "Virtualization" section in Windows Task Manager -> Performance -> CPU (it's located under "More details" if you don't see the Performance tab).
2. You have virtualization enabled - you have to enable it in your BIOS if you don't.
3. You have Windows 11 Pro - you can also use Windows 11 Home (also Windows 10 above certain version), but I cannot guarantee that provided scripts will work their magic.
4. You have Nvidia GPU - this is mandatory for current configuration. Support for AMD is presumably possible, but won't be added until such request shows up. Make sure you also have the newest drivers! Whole repository is based on CUDA 12 - you will be limited to GTX 900-series or higher.
5. You need admin access. These scripts use a PowerShell library that I've prepared, called WSLTools (handles automatic and interactive installation of WSL distributions from source), you need to have admin privileges to install this module.

How to use

After installation simply execute start.bat file to start the LLM app. You can open it under http://localhost:7860/?__theme=dark.

If you want to close the app - simply launch stop.bat, it will terminate the application and close the terminals.

Note! Keep in mind that stop.bat will terminate and remove all containers based on LLM webui image. If you have downloaded additional models while the application was running - they will have to be redownloaded again.

Installation

Automatic

Run install.bat in order to install the LLM. This will take a while - as long as you don't see red errors - everything's fine.

Manual

1. Install Windows 11
2. Install WSL from MS Store (https://www.microsoft.com/store/productId/9P9TQF7MRM4R)
3. Search for "Turn Windows features on or off" and enable "Hyper-V"
4. Set WSL to use v2: wsl --set-default-version 2
5. Install Linux distro of your choice (Ubuntu given as example): wsl --install Ubuntu
   1. Set up your username and password
6. (In distro command line) sudo sh -c 'echo "[boot]\nsystemd=true" > /etc/wsl.conf'
7. Check your distro name using wsl --list
8. Shutdown all distros wsl --shutdown and restart the one we're using wsl --distribution Ubuntu
9. Make sure you have nvidia drivers installed on Windows
10. Now open WSL. From now on, everything is executed from there.
11. Execute following scripts (installs cuda drivers):

    sudo apt-key del 7fa2af80
    wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
    sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
    sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/3bf863cc.pub
    sudo add-apt-repository 'deb https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/ /'
    sudo apt-get update
    sudo apt-get -y install cuda

12. Check if you're able to see your GPU in WSL: nvidia-smi
13. Install docker:

    curl https://get.docker.com | sh \
    && sudo systemctl --now enable docker

14. Prepare gpg keys to install nvidia-docker:

    distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
    && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
    && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list```

15. Now we can install it: sudo apt-get install -y nvidia-docker2
16. Restart docker service: sudo systemctl restart docker
17. Check if docker container also sees your GPU: sudo docker run --rm --gpus all nvidia/cuda:12.0.1-base-ubuntu22.04 nvidia-smi
18. Run ./build.sh from repo directory to build the container. You can uncomment depth, upscaler, inpainting and gfpgan from Dockerfile (first generated image) but it will take much more space - default installation is ~65GB total.
19. Run ./run.sh to start container. Open http://localhost:7860/?__theme=dark to access the webui - you can do so from Windows of course.

Sources

1. WizardLM HuggingFace
2. oobabooga webui
3. Nvidia Container Runtime
4. Ubuntu GPU acceleration on WSL2
5. MS WSL systemd
6. Nvidia WSL