Enabling GPU support for Odysseus

[Shirotabi]

I'm basically brand new to running an AI locally. I saw the recent video about Odysseus (like many newbies, no doubt) and decided to go for it. I'm running into a nauseating problem where I cannot seem to get Odysseus to recognize my GPU.

I got Docker for Desktop, I have WSL2, I have the Nvidia Container Toolkit. Integration with my default distro is enabled in Docker. I can see my graphics card in powershell and in ubuntu. nvidia-smi returns my graphics card details as expected.

But for some reason Odysseus does not recognize it. If I just type wsl in powershell without explicitly stating wsl -d Ubuntu , I start "docker-desktop" instead, where it seems that my GPU is also not recognized.

For reference, I also have this in my docker-compose.yml
ollama: image: ollama/ollama:latest ports: - "11434:11434" volumes: - ollama-data:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] restart: unless-stopped

I am a bit unsure what I'm missing and I feel like I made my GPU visible to every damn thing EXCEPT Odysseus. Can someone lend a hand please?

[DanteTheRA]

No many people are having this issue its not just you. dont know how to fix it just yet.

[ryslan25500-cloud]

FELIX will tell you in due time. Give the person time.

[ryslan25500-cloud]

I know you helped Felix create this piece of shit. that his house almost burned down 2-3 times. People should know their heroes. I see you're not going to move to rust at all.

[PardonMySkillz]

I suggest opening a Ubuntu dual boot on your PC and run it locally. WSL is a bit finicky to deal with.

[0wlan]

i've had the same problem where my 9070xt just wasnt being used at all if i run my models off ollama and docker. instead i switched running natively on windows with lm studios hosting locally from there instead where i can actually configure within lmstudios to offload to gpu. i doubt this is the best solution but it's my workaround for now and things are running well on the llm side at least.

[pdlzz]

I found as my only solution too.. however because it runs fresh everytime its doesnt remember any of my own data. bascially a new build everytime. Have you figured a workaround for this?

[Andrarch]

Are you asking about the LLM remembering your data? The LLM works by predicting the next text, based on the context you send in and the model, with a bunch of other fuzzy stuff going on in the middle to make them 'think', like randomizing token chosen, and looping.

But the model is basically just a frozen snapshot of all/whatever subset of human knowledge that they were able to compress into a weird sort of vectorized database. It knows nothing, and can learn nothing. At least in the conventional or human sense. If it knows or understands in some sense it's not the way a human person does.

What it has is a context, I'm not sure if lm studio caches/saves the context between passes, but basically you have to send in the context to the LLM each time you put in a request. Basically it's sending in everything you said to it before. That is how it 'knows' your data. When you open a new chat and prompt again it will never know what was in your other chats. It will only know what you put into the current context. Or what gets automatically put into the context.

It seems like a brain, it seems kind of smart, but it's not a person, and the way it thinks is weird. It's just trained to sound like a human.

[bitboody]

Are you sure it is an Odysseus problem and not a WSL one?

[Shirotabi]

I have a suspicion that it might have something to do with how Docker is set up, as opposed to WSL or Odysseus itself. Since the repo now addresses GPU detection in Cookbook explicitly and has some new scripts, I plan to do wipe and do a fresh install when I have time. If it works I'll update this thread.

[assk0401]

I have also tried to use WSL to use gpu but it was so hard I couldn't do it. I recommend a dual boot. It solves many problems.

[bigfurma-bot]

Yo!
I have 0 technical knowledge and spent 3 days to figure it out, (all hail Claude).
The standard process for Docker Desktop:

git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
cp .env.example .env  

docker compose up -d --build

Do just first 3 lines. Before you compose build with the 4th line, go to cd odysseus (folder you clonned you git to.)
Find docker-compose file.
Open it, find _PGID=${PGID:-1000}_ (line 65), add following lines in between _PGID=${PGID:-1000}_ and _chromadb:_

      # NVIDIA overlay (from docker/gpu.nvidia.yml).
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
    depends_on:
      searxng:
        condition: service_healthy
      chromadb:
        condition: service_started
    restart: unless-stopped
        # NVIDIA overlay (from docker/gpu.nvidia.yml).
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
       

Save file and compose with docker compose up -d --build

Voila! (hope you have nVidia card)

Alternatively in cd odysseus there are 2 additional files docker-compose.gpu-amd and docker-compose.gpu-nvidia.
The second one has same code Claude adviced me to add, so I suspect if you rename the one you need to base docker-compose file it would work too, thought I have not tried.

[Dafodilrat]

So guys I am keeping track of this issue currently, The issue currently lies in the base image lacking cuda support. This means that llama.cpp or other backends detects the gpu but fails to use it. The issue is currently a work in progress. You can keep track of this issue here #831.

Warning: Do not change the base image while on the main branch. There is a known issue where existing models are not detected in Cookbook after a rebuild and recreation. I am unsure if the fix for this has been merged into main yet.

[Dafodilrat]

Also note this is an issue you will face after you get nvidia-smi working in the docker container running odysseus.

you can use docker exec -it odysseus nvidia-smi to see if docker is able to detect gpu via container toolkit. (just spam tab after -it to get the container name).

[Dafodilrat]

Also in the official instruction after doing cp command go to .env file and uncomment this line COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml this will enable nvidia container tookit when using docker compose up command.

[ChiefHairyNPZ]

actually explains it in the read me under quick start

put the dl in ur user file make sure its just named Odesseus then do the three commands in terminal that r in the read me and run.

after an hour of being stuck at this step i finally got it going doing it this way.

im using docker and ollama.

[horribleCodes]

I've submitted a fix for the failing hardware detection problem on WSL hosts a few days ago: #2933
I haven't checked, but I imagine it's in the main branch by now. There's still some issues with actually serving a model, though.
If you don't mind losing out on some of the features from cookbook, the easiest workaround at the moment is to run an ollama or vLLM service with systemctl and connecting to it via API in "Settings > Add Models > API > Custom URL".

[ryslan25500-cloud]

#3255 reed

[Dafodilrat]

Literally has nothing to do with Odysseus.

[ryslan25500-cloud]

This is just words, but I have physics and already one precedent in my hands.

[ScuffedMallow]

this is a tutorial on how to workaround the GPU visibility problem and get better results than with the in container models.
#3691

[Cookiejunky]

You either:

1. Run host llm on llama.cpp or ollama and import the port to odysseus in docker - both work ok. - best way for multi PC's. Best CPU+GPU combo.
2. Compose with cuda image for nvidia or normally for amd (it works from what I know) - see my PR - single PC.
3. Default install + vlmm in Cookbook. (If your GPU is a swine in VRAM department)
4. Run on host in .venv/locally (not the best, but on a clean system is ok, due to security reasons during active development) - will run okay, if, you install everything on host.

[RlySkiz]

I thought rocm support for AMD GPU's had been added a while ago yet coming back from running ollama myself i tried it and it made my CPU usage shoot up like crazy while no GPU was used at all.
In cookbook it actually DID find my AMD GPU and correctly added rocm filter but apparently didn't install the model correctly with the ollama rocm version? Or at least didn't serve it correctly so that the GPU is actually utilized.