Help to get OpenHands working with local LLM

[grannymaster]

Hey,

I'm having some trouble getting OpenHands working with local ollama instance.

My whole setup is running on WSL in Ubuntu machine.

Ollama:
I installed ollama using the default command from their website:

curl -fsSL https://ollama.com/install.sh | sh

These are models I have installed:

ollama list
NAME                ID              SIZE      MODIFIED
qwen2.5-coder:7b    2b0496514337    4.7 GB    4 days ago
llama3:latest       365c0bd3c000    4.7 GB    6 weeks ago

Open WebUi:
Installied by following NetworkChuks's instructions, this is the docker container run command:

docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main

OpenHands
For OpenHands I used Docker compose:

name: openhands
services:
    openhands:
        stdin_open: true
        tty: true
        pull_policy: always
        environment:
            - SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.25-nikolaik
            - LOG_ALL_EVENTS=true
            - LLM_OLLAMA_BASE_URL="http://host.docker.internal:11434"
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - ~/.openhands-state:/.openhands-state
        ports:
            - 3000:3000
        extra_hosts:
            - host.docker.internal:host-gateway
        container_name: openhands-app
        image: docker.all-hands.dev/all-hands-ai/openhands:0.25

OpenHands provider settings:

One thing I'm not sure is the API key, as I've not set one up at any point, but according to things I found online, setting it to "ollama" should be good enough. This might be complitely wrong.

When I type my prompt for OpenHands, this the error I get:

RuntimeError: There was an unexpected error while running the agent. Please report this error to the developers. Your session ID is 6909e00c6e9c41d9a1660ae705ad26c7. Error type: APIConnectionError

[grannymaster]

Also tried this and 127.0.0.1

[enyst]

Please check out this, we've been trying to gather instructions that work, depending how you're running. Just in case it may matter (maybe it doesn't), you may want to try running with the command in the README in this repo.
https://docs.all-hands.dev/modules/usage/llms/local-llms

[grannymaster]

Sadly I'm getting the same error, I tried running the container using "Docker run" instead of compose.

Attempt #1

sudo docker run -it --rm --pull=always \
    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.25-nikolaik \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/.openhands-state:/.openhands-state \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
	--add-host host.docker.internal:host-gateway \
    -e LLM_OLLAMA_BASE_URL="http://host.docker.internal:11434" \
    docker.all-hands.dev/all-hands-ai/openhands:0.25

Attempt #2

sudo docker run -it --rm --pull=always \
    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.25-nikolaik \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/.openhands-state:/.openhands-state \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
	--add-host host.docker.internal:host-gateway \
    -e LLM_OLLAMA_BASE_URL="http://localhost:11434" \
    docker.all-hands.dev/all-hands-ai/openhands:0.25

[enyst]

Not sure what the problem is, but just double-checking: it does need http://host.docker.internal:11434 as base url in the UI Settings I think. And the model, yes, as you set it, with ollama/ and then the exact name reported by ollama list.

Could you please also double-check this section: https://docs.all-hands.dev/modules/usage/llms/local-llms#configuring-ollama-service-wsl-en
to enable Ollama service to be accessible.

[grannymaster]

Oh shit, I missed the the fixing of the WSL stuff.

Now tho when I prompt stuff I get this:

Parameter 'command' is expected to be one of ['view', 'create', 'str_replace', 'insert', 'undo_edit'].

EDIT: Seems to be an issue when using ollama\llama3:latest

[enyst]

That one is from the LLM itself. I'm afraid openhands needs a relatively powerful LLM that obeys instructions. I don't know the quirks of llama3, maybe it's temporary? I mean, you could perhaps tell the LLM to pay attention to the exact definitions it needs to follow?

[grannymaster]

Could be, but it's working now! Thanks for your help!

[LonglifeIO]

@grannymaster how did you get this working? I tried with the latest docker command you shared. Using deepseek and qwen it doesn't seem to work

[grannymaster]

I've stopped using this setup as I had no need for it, but If remember correctly that one of the wiki pages linked by @enyst had a section about WSL and maybe something related to allowing network traffic to reach the container. I think it's what I was referencing with this:

Oh shit, I missed the the fixing of the WSL stuff.

But I'm not 100% sure. Maybe @enyst can help you out too?

[LonglifeIO]

Ok awesome, thanks for the quick reply. I'll look into it, otherwise I might look at another option instead of OpenHands

[enyst]

@NephilimBray

The doc with WSL troubleshooting and Ollama setup was this:
https://github.com/All-Hands-AI/OpenHands/blob/824b81deeb16c43cf0340a0850dc9cdef59da4c3/docs/modules/usage/llms/local-llms.md

If it's not helping (and it's becoming outdated now), could you perhaps open a new issue to tell what setup do you use and what is the error? We have a template for bugs that helps us get information we need to support you

[danielguerra69]

@grannymaster , I was investigating the use of OpenHands and Ollama and the host you used triggered me : host.docker.internal
On a Mac use this host from docker to local your ollama : docker.for.mac.localhost

[danielguerra69]

Did check the linux way:
Here's a breakdown of the methods:

1. Using host.docker.internal:
   Explanation:
   This hostname is specifically designed for Docker and resolves to the host's IP address. It's the recommended approach for most scenarios.
   Linux Setup:
   For Linux systems (Docker v20.10 and above), you need to add the --add-host flag:
   Code

   docker run --add-host=host.docker.internal:host-gateway -it ubuntu bash
   Usage: Once the container is running, you can use http://host.docker.internal:port (replace port with the actual port) to access services on the host.

2. Using the Host Network:
   Explanation: This option makes the container share the host's network namespace, effectively making localhost inside the container the same as localhost on the host.
   Usage:
   Code

   docker run --network=host -it ubuntu bash
   Considerations: While simple, this can expose the container's internal ports to the host and vice-versa, so use it with caution.

3. Using the Host's IP Address:
   Explanation: You can find the host's IP address (e.g., using hostname -I and then use that IP address in your container.
   Usage:
   Code

   Get the host's IP address

   HOST_IP=$(hostname -I | awk '{print $1}')

   Run the container, accessing the host via the IP

   docker run -it -e HOST_IP=$HOST_IP ubuntu bash

   Inside the container, you can then access the host at $HOST_IP:port

Considerations: This method is less convenient as you need to know the host's IP beforehand and handle potential IP changes.
4. Docker Compose:
host.docker.internal: For Linux with Docker Compose, add this to the container definition:
Code

extra_hosts:
  - "host.docker.internal:host-gateway"

Host Network: You can use the network_mode: host option in your docker-compose.yml file.