Openai Compatible API

[Dormiveglia-elf]

Any idea to invoke Openai Compatible API with specific base_url? (both LLM and Embedding model)

[stellasia]

Hi @Dormiveglia-elf ,

You can pass the base_url to the OpenaAILLM and OpenAIEMbeddings classes like this:

llm = OpenAILLM(
    model_name="gpt-4o",
    base_url="https://...",
)

and same for embedder.

Our classes are wrappers around the openai Python package, so all init parameters can be propagated this way.

[michela]

llm = OpenAILLM(
    model_name=LLM_MODEL, 
    base_url=OLLAMA_ENDPOINT
)

This doesn't appear to work if using http URL

packages/neo4j_graphrag/llm/openai_llm.py", line 115, in invoke
    raise LLMGenerationError(e)
neo4j_graphrag.exceptions.LLMGenerationError: 404 page not found

See undocumented workaround reported here #274

[stellasia]

Hi again @michela :)

A 404 error suggests a wrong config. Which URL are you using? Have you tried to curl it? Have you tried the suggestion in the other issue?

[michela]

Hi again. Thanks for the quick reply!

Yes, curl find and via test below all using my .env (with being a real IP)

OLLAMA_ENDPOINT=http://<IP>:11434

This test code code below works fine (and with also with Ollama host in #274) with same environment variable reference. Only get 404 in this context

from ollama import Client
client = Client(
  host=OLLAMA_ENDPOINT,
)
response = client.chat(model='llama3.2:3b', messages=[
  {
    'role': 'user',
    'content': 'Why is the sky blue?',
  },
])
print(response)

[stellasia]

For OpenAI, I think you need to append v1 at the end of the URL.

[stellasia]

CLosing this issue as examples were added to use a base_url / host parameters in OpenAI/Ollama classes.