How I managed to run Odysseus in WSL2

[rmzmrnn]

My PC hardware

		Remarks
CPU	AMD Ryzen 7 5700X 8-Core Processor
GPU	NVIDIA GeForce RTX 3050	NVIDIA-SMI 610.43.02 KMD Version: 610.47 CUDA UMD Version: 13.3 8GB VRAM
Memory	16GB RAM
Storage	1TB
OS	Windows 25H2 [OS Build 26200.8524]

WSL2 Initialization

1. Make sure that you have WSL2 installed in your computer. If not, go to this link: How to install Linux on Windows with WSL. Ubuntu 24.04.1 LTS is the distro that I am using
2. Open WSL2 once it is installed and do the initializations shown on the terminal. Once it is done, run this command

sudo apt update
sudo apt upgrade

3. You can also run sudo apt autoremove for good measure.
4. Run nvidia-smi. This will check if WSL2 can detect your GPU. You should see an output like this:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.43.02              KMD Version: 610.47        CUDA UMD Version: 13.3     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3050        On  |   00000000:06:00.0  On |                  N/A |
| 32%   39C    P5            N/A  /  115W |     713MiB /   8192MiB |     21%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Notes:

• If it didn't work, try updating your GPU drivers (❗NOT IN WSL2 use the NVIDIA App for ease-of-use❗). Then, restart WSL2 (Let me know if there are any further problems. So far, I haven't had any issues with nvidia-smi).
• If you want more information, go to CUDA on WSL User Guide

Miniconda installation

This is only my personal preference. I use miniconda to manage my Python projects and environments.

1. To install Miniconda, follow this instruction: Miniconda Linux Install
2. Once installed, create an environment:

conda create -n odysseus python=3.11

3. Activate the environment you created:

conda activate odysseus

Running Odysseus

1. Fork the Odysseus repo and clone your forked repo using this command git clone <url>
2. Go to the odysseus folder: cd odysseus
3. Run these commands:

pip install -r requirements.txt
pip install vllm==0.21.0 --force-reinstall # <--- This will be further explained in "Using Odysseus" section
python setup.py
python -m uvicorn app:app --host 127.0.0.1 --port 7000

※ If you noticed, there is the command pip install vllm==0.21.0 --force-reinstall this will be explained in "Using Odysseus" section at Step 3

You should see this in your terminal if ran successfully:

Using Odysseus

1. Enter http://127.0.0.1:7000/ in your web browser
2. Go to Cookbook -> Download. There will be a list of models that you can run based on the hardware you have. If the list didn't appear, hit the Scan/Rescan button.

3. Next is to download the model. In my case, I downloaded this model cyankiwi/LFM2-8B-A1B-AWQ-4bit

Note: This is the reason why I installed vllm==0.21.0 because the model uses vllm as a library for LLM inference and serving. Also, this specific version 0.21.0 works on my environment. As of this writing, the latest version of vllm causes errors on my end but please let me know if the latest version of vllm works.

4. Next, go to Cookbook -> Serve and look for your downloaded model.

Note: Because of the limitations of my hardware, I set the Context = 10000 and GPU Mem = 0.80

5. Press Launch then wait for it to be ready

6. Go to New Chat and select the running model

7. You can now chat with Odysseus! 🎉🎉🎉

Other Notes

• If you've made it this far, thank you for reading! I really appreciate it 🙇. I hope you were able to get Odysseus running successfully.
• These instructions did not consider other features. The only purpose is to get started chatting with Odysseus.
• I am yet to explore other functions and getting my hands dirty with this project.

[KangaZero]

Very cool! Thanks for sharing. I am also using WSL2 but with NixOS. I get Odysseus running using a flake (something like a Docker container file or pip's requirement.txt + venv if unaware). You do not actually need NixOS the distro, just Nix the package manager. Very cool if you never tried declaratively configuring your machine, but also very painful at the same time!

My setup and instructions are listed in the repos below:
wsl-config
Odysseus-nix - Not proud but 100% vibe coded