RAGFlow 1.0: adding an OpenAI-compatible gateway as an "OpenAI-API-Compatible" provider instance (custom model IDs, context length, tool calling, defaults)

[apigoto]

I work on the APIGOTO team. APIGOTO is an independent OpenAI-compatible gateway, not affiliated with OpenAI, Anthropic, Google or InfiniFlow. Below is a step-by-step for the provider-instance UI in the 1.0 line (current docs/guides/models/llm_api_key_setup.md). It works for any OpenAI-compatible endpoint, local or hosted.

0. Check what the endpoint actually serves

curl https://api.china-aiapi.com/v1/models -H "Authorization: Bearer $API_KEY"

Note the exact IDs of the chat model you want (MODEL_ID). If the endpoint has one, also note the embedding model (EMBEDDING_MODEL_ID). RAGFlow needs a default LLM and a default embedding model.

1. Create the instance

• User settings → Model providers → in Available models, pick OpenAI-API-Compatible.
• Instance name: anything, for example gateway-prod. Instances let you keep a test and a production gateway, or a gateway and a local server, side by side.
• Base URL: https://api.china-aiapi.com/v1. The docs warn that with a compatible API you must check whether the path needs /v1. For this kind of endpoint it does.
• API Key: your key.

2. Add models

If the model isn't in the discovered list, use Add custom model:

• Model name: exactly MODEL_ID as returned by /models. The docs say a mismatch means RAGFlow "may not be able to identify the model correctly during calls".
• Model type: LLM (and a second entry of type Embedding for EMBEDDING_MODEL_ID, if you use one from the gateway).
• Max context length: only fill it in if you know the provider's real limit. Leave the default otherwise rather than guessing high.
• Tool calling: enable it only if that model supports tool calls on this endpoint. Agents with tools depend on it, and the docs say not to enable it for models that don't.

3. Verify, then set defaults

• Click Verify per model, or Verify all models.
• Then set the Default LLM and Default embedding model (the two the docs list as minimum), plus rerank if you have one.

4. Embeddings: decide before you parse

The embedding model determines the vectors stored for each dataset. If your gateway doesn't serve embeddings, keep a local embedding model as default and use the gateway only for the LLM. Switching the embedding model after documents are parsed generally means re-parsing them, so choose first.

Troubleshooting

• Verify fails with 404: /v1 missing or doubled.
• Verify fails with 401: key pasted with whitespace, or saved on another instance.
• Verify OK, but chats fail on long documents: the Max context length you entered is higher than what the model actually accepts. Lower it.
• Agent never calls tools: Tool calling not enabled on that model entry, or the model doesn't support it.
• Docker self-hosting with a local OpenAI-compatible server: localhost inside the container isn't your host. Use a host-reachable address. A hosted gateway URL is unaffected.

On the gateway side: one base URL for several model families, automatic failover on upstream errors (at most 3 retries per request on another route), a per-request log, pay-as-you-go, BYOK: www.china-aiapi.com

Has anyone run separate instances for chat and for agent workloads? I'm curious how you split defaults.

[Yannnnnnny]

Thanks for sharing this detailed guide with the RAGFlow community! We appreciate your contribution!

[apigoto]

Thanks @Yannnnnnny, glad it's useful! If anyone runs into trouble with the provider setup (custom model IDs, context length, tool calling), feel free to ask here. (Disclosure: I work on the APIGOTO team.)