What happened?
In a completely air-gapped environment, the /vector_stores/VS_ID/search endpoint ignores the embedding_model setting in config.yaml when the specified model uses a custom provider (in my case, hosted_vllm).
Instead of using the configured local model, LiteLLM falls back to its default behavior and attempts to connect to api.openai.com to generate embeddings. This call is impossible in an air-gapped setup, causing the request to fail.
This suggests that the vector store logic may not correctly recognize or handle custom model providers, causing it to ignore the configuration and revert to a hardcoded default.
How to Reproduce
Set up LiteLLM Proxy in an air-gapped Kubernetes or OpenShift environment.
Configure a local embedding model served via a custom OpenAI-compatible endpoint (e.g., vLLM). In the config.yaml, define this model using the hosted_vllm prefix.
Configure a vector store (e.g., pg_vector) and link it to the local model using the embedding_model parameter, referencing the model_name from step 2.
Send a POST request to the /vector_stores/VS_ID/search endpoint.
Expected Behavior
LiteLLM should honor the embedding_model: "local-embedding" configuration.
It should make a network request to the internal api_base (https://qwen3-embedding-8b...) to generate embeddings for the search query.
It should not make any external network calls to api.openai.com.
Actual Behavior
LiteLLM ignores the configuration for the hosted_vllm model.
It attempts to connect externally to api.openai.com:443.
The request fails with the error: litellm.APIConnectionError: openai - Cannot connect to host api.openai.com:443 ssl:default [Name or service not known].
Environment
Deployment: Air-gapped OpenShift Cluster
LiteLLM Version: Latest Docker Image (as of Oct 2025)
Vector Store: pg_vector -- litellm_pgvector
Embedding Model Provider: Custom hosted_vllm endpoint (OpenAI-compatible vLLM service)
Debugging Steps Taken
Confirmed the LiteLLM pod had restarted and loaded the latest ConfigMap multiple times.
Verified that the LiteLLM pod has direct network connectivity to the embedding model's internal service URL (https://qwen3-embedding-8b...). A python urllib script run from within the LiteLLM pod successfully connects to the endpoint, proving it's not a network or SSL issue.
Setting the OPENAI_API_VERIFY_SSL_CERTS=False environment variable had no effect, which is expected since the call is being incorrectly routed to the public OpenAI URL, not the internal one.
Potential Workaround Found: If I "disguise" the model as an openai model (e.g., model: openai/my-model) while keeping the local api_base, LiteLLM appears to respect the configuration. This strongly suggests the issue is tied to how LiteLLM's vector store feature handles non-standard provider names like hosted_vllm.
Relevant log output
Are you a ML Ops Team?
No
What LiteLLM version are you on ?
main-v1.77.5-stable
Twitter / LinkedIn details
No response
What happened?
In a completely air-gapped environment, the /vector_stores/VS_ID/search endpoint ignores the embedding_model setting in config.yaml when the specified model uses a custom provider (in my case, hosted_vllm).
Instead of using the configured local model, LiteLLM falls back to its default behavior and attempts to connect to api.openai.com to generate embeddings. This call is impossible in an air-gapped setup, causing the request to fail.
This suggests that the vector store logic may not correctly recognize or handle custom model providers, causing it to ignore the configuration and revert to a hardcoded default.
How to Reproduce
Set up LiteLLM Proxy in an air-gapped Kubernetes or OpenShift environment.
Configure a local embedding model served via a custom OpenAI-compatible endpoint (e.g., vLLM). In the config.yaml, define this model using the hosted_vllm prefix.
Configure a vector store (e.g., pg_vector) and link it to the local model using the embedding_model parameter, referencing the model_name from step 2.
Send a POST request to the /vector_stores/VS_ID/search endpoint.
Expected Behavior
LiteLLM should honor the embedding_model: "local-embedding" configuration.
It should make a network request to the internal api_base (https://qwen3-embedding-8b...) to generate embeddings for the search query.
It should not make any external network calls to api.openai.com.
Actual Behavior
LiteLLM ignores the configuration for the hosted_vllm model.
It attempts to connect externally to api.openai.com:443.
The request fails with the error: litellm.APIConnectionError: openai - Cannot connect to host api.openai.com:443 ssl:default [Name or service not known].
Environment
Deployment: Air-gapped OpenShift Cluster
LiteLLM Version: Latest Docker Image (as of Oct 2025)
Vector Store: pg_vector -- litellm_pgvector
Embedding Model Provider: Custom hosted_vllm endpoint (OpenAI-compatible vLLM service)
Debugging Steps Taken
Confirmed the LiteLLM pod had restarted and loaded the latest ConfigMap multiple times.
Verified that the LiteLLM pod has direct network connectivity to the embedding model's internal service URL (https://qwen3-embedding-8b...). A python urllib script run from within the LiteLLM pod successfully connects to the endpoint, proving it's not a network or SSL issue.
Setting the OPENAI_API_VERIFY_SSL_CERTS=False environment variable had no effect, which is expected since the call is being incorrectly routed to the public OpenAI URL, not the internal one.
Potential Workaround Found: If I "disguise" the model as an openai model (e.g., model: openai/my-model) while keeping the local api_base, LiteLLM appears to respect the configuration. This strongly suggests the issue is tied to how LiteLLM's vector store feature handles non-standard provider names like hosted_vllm.
Relevant log output
Are you a ML Ops Team?
No
What LiteLLM version are you on ?
main-v1.77.5-stable
Twitter / LinkedIn details
No response