[Bug]: [Allow /models to return correct models for custom wildcard prefixes] feature didn't work

[changchiyou]

What happened?

#11784 (comment)

I got this {'data': [{'id': '*', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'} with the following setting:

model_list:
  - model_name: "*"
    litellm_params:
      model: openai/*
      api_key: os.environ/OPENAI_API_KEY
      api_base: os.environ/OPENAI_API_BASE
litellm_settings:
  drop_params: true
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]
  callbacks: custom_callbacks.proxy_handler_instance
  check_provider_endpoint: true

general_settings:
  master_key: "os.environ/MASTER_KEY"

Image Version:

➜  openwebui git:(main) ✗ docker image ls | grep litellm
ghcr.io/berriai/litellm                                                                        main-latest   007e9a726cf2   35 hours ago    2.17GB

Relevant log output

INFO:     127.0.0.1:48536 - "GET /health/readiness HTTP/1.1" 200 OK
09:38:25 - LiteLLM Proxy:DEBUG: hanging_request_check.py:148 - Checking for hanging requests....
INFO:     127.0.0.1:60096 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     127.0.0.1:40044 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     127.0.0.1:59244 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     Shutting down
INFO:     Waiting for application shutdown.
09:39:34 - LiteLLM Proxy:INFO: proxy_server.py:518 - Shutting down LiteLLM Proxy Server
INFO:     Application shutdown complete.
INFO:     Finished server process [1]
   ██╗     ██╗████████╗███████╗██╗     ██╗     ███╗   ███╗
   ██║     ██║╚══██╔══╝██╔════╝██║     ██║     ████╗ ████║
   ██║     ██║   ██║   █████╗  ██║     ██║     ██╔████╔██║
   ██║     ██║   ██║   ██╔══╝  ██║     ██║     ██║╚██╔╝██║
   ███████╗██║   ██║   ███████╗███████╗███████╗██║ ╚═╝ ██║
   ╚══════╝╚═╝   ╚═╝   ╚══════╝╚══════╝╚══════╝╚═╝     ╚═╝
INFO:     Started server process [1]
INFO:     Waiting for application startup.
09:39:48 - LiteLLM Proxy:DEBUG: proxy_server.py:554 - litellm.proxy.proxy_server.py::startup() - CHECKING PREMIUM USER - False
09:39:48 - LiteLLM Proxy:DEBUG: litellm_license.py:102 - litellm.proxy.auth.litellm_license.py::is_premium() - ENTERING 'IS_PREMIUM' - LiteLLM License=None
09:39:48 - LiteLLM Proxy:DEBUG: litellm_license.py:111 - litellm.proxy.auth.litellm_license.py::is_premium() - Updated 'self.license_str' - None
09:39:48 - LiteLLM Proxy:DEBUG: proxy_server.py:567 - worker_config: {"model": null, "alias": null, "api_base": null, "api_version": "2024-07-01-preview", "debug": false, "detailed_debug": false, "temperature": null, "max_tokens": null, "request_timeout": null, "max_budget": null, "telemetry": true, "drop_params": false, "add_function_to_prompt": false, "headers": null, "save": false, "config": "/app/config.yaml", "use_queue": false}
09:39:48 - LiteLLM Proxy:DEBUG: proxy_server.py:1966 -  setting litellm.drop_params=True
09:39:48 - LiteLLM Proxy:DEBUG: callback_utils.py:22 - initializing callbacks=custom_callbacks.proxy_handler_instance on proxy
09:39:48 - LiteLLM Proxy:DEBUG: callback_utils.py:285 -  Initialized Callbacks - [<custom_callbacks.MyCustomHandler object at 0x72dff0e59fd0>] 
09:39:48 - LiteLLM Proxy:DEBUG: proxy_server.py:1966 -  setting litellm.check_provider_endpoint=True
09:39:48 - LiteLLM Proxy:DEBUG: proxy_server.py:2270 - _alerting_callbacks: {'master_key': 'sk-xxx'}
09:39:48 - LiteLLM Router:DEBUG: router.py:5051 - 
Initialized Model List ['*']
09:39:48 - LiteLLM Router:INFO: router.py:693 - Routing strategy: simple-shuffle
09:39:48 - LiteLLM Router:DEBUG: router.py:559 - Intialized router with Routing strategy: simple-shuffle
Routing enable_pre_call_checks: False
Routing fallbacks: None
Routing content fallbacks: None
Routing context window fallbacks: None
Router Redis Caching=None
09:39:51 - LiteLLM Proxy:DEBUG: proxy_server.py:641 - prisma_client: None
09:39:51 - LiteLLM Proxy:DEBUG: hanging_request_check.py:148 - Checking for hanging requests....
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
#------------------------------------------------------------#
#                                                            #
#           'I get frustrated when the product...'            #
#        https://github.com/BerriAI/litellm/issues/new        #
#                                                            #
#------------------------------------------------------------#
 Thank you for using LiteLLM! - Krrish & Ishaan
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
 Initialized Success Callbacks - ['langfuse'] 
 Initialized Failure Callbacks - ['langfuse'] 
LiteLLM: Proxy initialized with Config, Set models:
    *
INFO:     127.0.0.1:54620 - "GET /health/readiness HTTP/1.1" 200 OK
09:40:15 - LiteLLM Proxy:DEBUG: model_checks.py:121 - ALL KEY MODELS - 0
09:40:15 - LiteLLM Proxy:DEBUG: model_checks.py:153 - ALL TEAM MODELS - 0
INFO:     172.24.0.6:35996 - "GET /v1/models HTTP/1.1" 200 OK
09:40:15 - LiteLLM Proxy:DEBUG: model_checks.py:121 - ALL KEY MODELS - 0
09:40:15 - LiteLLM Proxy:DEBUG: model_checks.py:153 - ALL TEAM MODELS - 0
INFO:     172.24.0.6:36012 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:42326 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     127.0.0.1:47434 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     127.0.0.1:36986 - "GET /health/readiness HTTP/1.1" 200 OK

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

main-latest(007e9a726cf2)

Twitter / LinkedIn details

No response

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[badjware]

I have hit the exact same issue you described here. Hopefully your PR will be merged soon.

[changchiyou]

Hi @krrishdholakia , I think I've found the root cause of the issue. I noticed that get_models in gpt_transformation.py parses the URL with httpx and then manually reassembles the scheme, host, and port instead of using the URL directly. I'm not sure why it does this, but it results in the path being stripped out if the user's api_base includes one:

litellm/litellm/llms/openai/chat/gpt_transformation.py

Lines 708 to 722 in 4fffd33

	if api_base is None:
	api_base = "https://api.openai.com"
	if api_key is None:
	api_key = get_secret_str("OPENAI_API_KEY")
	# Strip api_base to just the base URL (scheme + host + port)
	parsed_url = httpx.URL(api_base)
	base_url = f"{parsed_url.scheme}://{parsed_url.host}"
	if parsed_url.port:
	base_url += f":{parsed_url.port}"
	response = litellm.module_level_client.get(
	url=f"{base_url}/v1/models",
	headers={"Authorization": f"Bearer {api_key}"},
	)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.