[Feature]: support model discovery with custom provider

[cornmail]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

The Feature

Here is an example of how the desired config.yaml could look:

model_list:
  - model_name: my-custom-service/*  # Use a wildcard for the custom provider group
    litellm_params:
      model: custom_openai/* # The litellm provider identifier
      api_key: os.environ/MY_CUSTOM_PROVIDER_KEY
      api_base: "http://my-custom-provider-url.com/v1" # The endpoint for the custom provider

litellm_settings:
  check_provider_endpoint: true # Enable model discovery

With this configuration, a call to the LiteLLM proxy's /v1/models endpoint should trigger a discovery call to http://my-custom-provider-url.com/v1/models and return the result in the expected format.

This enhancement would significantly improve the integration of LiteLLM with the rapidly growing ecosystem of self-hosted and third-party OpenAI-compatible APIs.

Thank you for considering this feature request

Motivation, pitch

The new Model Discovery feature is an excellent addition for dynamically fetching a list of available models from a provider when using a wildcard in the configuration (e.g., xai/, openai/). This works well for the list of officially supported providers.

However, this functionality does not seem to extend to custom providers, specifically those that are OpenAI-compatible and configured using the custom_openai provider type. When using a custom OpenAI-compatible endpoint (like a private VLLM server, another LLM proxy, or a niche provider), we are still required to manually list every single model ID in the config.yaml. This is not scalable and requires manual configuration updates whenever the upstream provider adds or removes a model.

What part of LiteLLM is this about?

Proxy

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

Twitter / LinkedIn details

No response

[Sameerlite]

@cornmail Wouldn't this work: https://docs.litellm.ai/docs/contributing/adding_openai_compatible_providers?
You just need to add your model in the provider json since it is openai compatble, then the config you share should automatically work

[cornmail]

Hi @Sameerlite

I tried, but I'm not able to load my models:

11:59:37 - LiteLLM Router:ERROR: router.py:5648 - Error creating deployment: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=myprovider/*
 Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers, ignoring and continuing with other deployments.
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5638, in _create_deployment
    deployment = self._add_deployment(deployment=deployment)
  File "/usr/lib/python3.13/site-packages/litellm/router.py", line 5829, in _add_deployment
    ) = litellm.get_llm_provider(
        ~~~~~~~~~~~~~~~~~~~~~~~~^
        model=deployment.litellm_params.model,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        ),
        ^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/get_llm_provider_logic.py", line 476, in get_llm_provider
    raise e
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/get_llm_provider_logic.py", line 453, in get_llm_provider
    raise litellm.exceptions.BadRequestError(  # type: ignore
    ...<8 lines>...
    )
litellm.exceptions.BadRequestError: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=myprovider/*
 Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/provider

Here is my providers.json:

{
  "myprovider": {
    "base_url": "http://myprovider:8080/v1",
    "api_key_env": "MYPROVIDER_API_KEY"
  }
}

And my config.yml:

model_list:
  - model_name: "myprovider/*"
    litellm_params:
      model: "myprovider/*"
      api_base: http://myprovider:8080/v1
      api_key: "os.environ/MYPROVIDER_API_KEY"

LiteLLM version : v1.80.15

[rally12]

+1 This happens when using litellm as proxy in front of vllm and the loaded model in vllm is changing every now and then. The expectation is that litle llm follows/forwards the model name without having to edit the config.

[mamorett]

+1

[mkresse]

+1 — same need with litellm_proxy/* (LiteLLM in front of LiteLLM):

  model_list:
    - model_name: "*"
      litellm_params:
        model: "litellm_proxy/*"
        litellm_credential_name: "upstream"
  litellm_settings:
    check_provider_endpoint: true

/v1/chat/completions forwards correctly, but /v1/models returns nothing from upstream.

[thefounderofantirome]

For this custom-provider discovery case, I would split the problem into two checks:

1. Provider resolution: confirm whether LiteLLM resolves myprovider/* before it tries wildcard expansion. The error suggests provider lookup happens first, then discovery never gets a chance to call /v1/models.
2. Upstream discovery shape: call the upstream /v1/models with the same auth header LiteLLM would use, then verify the returned IDs can be mapped into LiteLLM aliases without manual per-model config.

For a short-term workaround, a small startup script can fetch the upstream /v1/models, generate the model_list entries, then launch LiteLLM with a concrete config. That avoids manually editing the config every time the upstream model set changes.

For LiteLLM-in-front-of-LiteLLM, I would also test whether the upstream /v1/models endpoint requires the same bearer token as /v1/chat/completions, because completions can work while discovery silently returns an empty model list.

If anyone needs a quick BYO-key config/debug pass for this custom OpenAI-compatible discovery setup, I can help trace the provider resolver and generate a working config.