OpenAILike for OpenAI-compatible gateways: the two flags that matter (is_chat_model, is_function_calling_model) and the errors you get without them

[apigoto]

I work on the APIGOTO team. APIGOTO is an independent OpenAI-compatible gateway, not affiliated with OpenAI, Anthropic or Google. Nothing below is specific to us: it applies to vLLM, LM Studio, LiteLLM or any hosted endpoint that speaks /v1/chat/completions. The point is three errors that keep showing up in issues when people point LlamaIndex at a non-OpenAI base URL.

Why not OpenAI(api_base=...)?

llama_index.llms.openai.OpenAI looks up the model name in its own table to get the context window and capabilities. With a gateway model ID it fails with Unknown model ... Please provide a valid OpenAI model name. OpenAILike skips that lookup, and you declare the metadata yourself.

Install

pip install llama-index-llms-openai-like

The LLM

import os
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="MODEL_ID",                        # exactly as listed by GET /v1/models
    api_base="https://api.china-aiapi.com/v1",
    api_key=os.environ["GATEWAY_API_KEY"],
    is_chat_model=True,                      # default False
    is_function_calling_model=True,          # only if the model really supports tools
    context_window=CONTEXT_WINDOW,           # default 3900, set the model's real value
    max_retries=1,
    timeout=60,
)
print(llm.complete("Say hi").text)

Error 1: 404 on /completions

is_chat_model defaults to False. Then even llm.chat(...) flattens your messages into a prompt and calls the legacy /completions endpoint, which most gateways and chat-only models don't serve. Set is_chat_model=True.

Error 2: ValueError: LLM must be a FunctionCallingLLM

FunctionAgent checks llm.metadata.is_function_calling_model before its first step. With the default False, it refuses to run:

from llama_index.core.agent.workflow import FunctionAgent

def multiply(a: float, b: float) -> float:
    """Multiply two numbers."""
    return a * b

agent = FunctionAgent(tools=[multiply], llm=llm, system_prompt="Use tools for arithmetic.")
response = await agent.run("What is 21.5 * 4?")
print(response)

If the model has no reliable tool calling, leave the flag False and use ReActAgent instead. It drives tools through plain text, so it doesn't need function-calling support.

Error 3: silent truncation

With context_window left at 3900, the prompt helper packs retrieved chunks for a 3900-token window. Nothing fails, but answers ignore most of your context. Set the model's real window.

Embeddings (only if your endpoint serves them)

Check /v1/models for an embedding model first. If there is none, keep a local embedding model and use the gateway only for the LLM.

pip install llama-index-embeddings-openai-like

from llama_index.core import Settings
from llama_index.embeddings.openai_like import OpenAILikeEmbedding

Settings.llm = llm
Settings.embed_model = OpenAILikeEmbedding(
    model_name="EMBEDDING_MODEL_ID",         # note: model_name, not model
    api_base="https://api.china-aiapi.com/v1",
    api_key=os.environ["GATEWAY_API_KEY"],
    embed_batch_size=16,
)

Two details: it takes model_name (passing model raises an error), and its max_retries defaults to 10, compared with 3 for the LLM. Lower it if your gateway already retries.

Retries

Our gateway retries a failed upstream call on another route, at most 3 times. LlamaIndex retries on top of that, so max_retries=1 on the client keeps worst-case latency predictable. Per-request logs on the gateway side show which retry actually answered. Pay-as-you-go, BYOK supported: www.china-aiapi.com

Has anyone hit a case where OpenAILikeResponses behaves better than OpenAILike against a third-party endpoint?