LiteLLM connection issue when deployed on another server

[kikohs]

I have successfully deployed a ollama model via LiteLLM on another machine.

When using guidance, I can't find a way to set the api_base other than localhost, see screenshot:

The correct server address should be 192.168.1.42:11434 instead of localhost.

And the error:

Thanks for your help

[kikohs]

Solution is simply to capture api_base and pass it along to litellm in _lite_llm.py

[linkedlist771]

Hi, it looks like your are trying to usue the guidance remotely. I am also want to do that but find that the python package lib of the guidance has not been updated yet. I wonder how to set up the server and the client in this issue as you said.

[kikohs]

I used ollama on a remote machine in server mode, then simply the LiteLLM package with the modifications I posted with the remote API using api_base.

[linkedlist771]

Your answer did solve my problem, but the documentation for guidance is poorly written. I couldn't find any mention of the LiteLLM remote before your explanation.