Repository navigation
OpenAIChatGenerator against a non-OpenAI base URL: api_base_url vs OPENAI_BASE_URL, stacked retries, and keeping keys out of pipeline YAML #13626
apigoto
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I work on the APIGOTO team. APIGOTO is an independent OpenAI-compatible gateway, not affiliated with OpenAI, Anthropic, Google or deepset. These notes apply to any OpenAI-compatible endpoint (a gateway, vLLM, LM Studio, a corporate proxy). Everything below comes from reading
haystack/components/generators/chat/openai.pyonmain(v3.4.0).1. Check the endpoint first
curl https://api.china-aiapi.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY"Copy a model ID from that list exactly. It is
MODEL_IDbelow.2. Two ways to set the base URL
api_base_url=is passed to the OpenAI client asbase_url.None, the OpenAI SDK falls back to theOPENAI_BASE_URLenvironment variable. The olderOPENAI_API_BASEname is not read. When a pipeline "ignores" your gateway, this is usually why.3. Retries stack
The generator defaults to
max_retries=5(orOPENAI_MAX_RETRIES) andtimeout=30seconds (orOPENAI_TIMEOUT). If your gateway also retries upstream errors, one failing call can turn into many upstream attempts and a long wait. Pick one layer to own retries:max_retries=1(or 0) in Haystack.Raise
timeoutfor long generations. 30 seconds is short for big outputs.4. Keep the key out of
pipeline.dumps()Secret.from_env_var(...)serializes as a reference to the variable name, not the value. A pipeline YAML committed to git then contains no key.Secret.from_token(...)is documented as "Cannot be serialized", so a pipeline using it can't be dumped at all.5. Embeddings are a separate decision
OpenAITextEmbedder/OpenAIDocumentEmbeddertake the sameapi_base_url. Only point them at the gateway if it actually serves an embedding model you have listed in/models. Otherwise keep a local embedder and use the gateway for generation only. Indexing and querying must use the same embedding model either way.Troubleshooting
404:/v1missing or doubled inapi_base_url.401: the env var isn't set in the process that runs the pipeline (Hayhooks, a worker, a notebook kernel started earlier)./modelsexactly, provider prefixes included.max_completion_tokensrejected: some backends still expectmax_tokensingeneration_kwargs.tools=.On the gateway side: one base URL for several model families, automatic failover on upstream errors (at most 3 retries per request on another route), a per-request log, pay-as-you-go, BYOK: www.china-aiapi.com
If you run Haystack through a proxy in production, which layer owns retries in your setup?
All reactions