[Feature]: support enabling "/responses to /chat/completions Bridge" on openai (llama.cpp) models

[RuthgerD]

The Feature

Make the following:
https://docs.litellm.ai/docs/response_api#calling-non-responses-api-endpoints-responses-to-chatcompletions-bridge

toggleable on openai/ models

Motivation, pitch

llama.cpp does not have support for /completions, but litellm thinks it does due to it being added as a openai model. It would be great if we can opt in to the bridge:
https://docs.litellm.ai/docs/response_api#calling-non-responses-api-endpoints-responses-to-chatcompletions-bridge

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

Twitter / LinkedIn details

No response

[ishaan-jaff]

@RuthgerD this should already work - please try it

[RuthgerD]

@ishaan-jaff In what way should this work? is there a setting available?

using 1.74.15, litellm still forwards /v1/responses to llama.cpp

litellm test key page using /v1/responses:

Error fetching response:Error: 404 litellm.NotFoundError: NotFoundError: OpenAIException - {"error":{"code":404,"message":"File Not Found","type":"not_found_error"}}. Received Model Group=

litellm logs:

INFO:     10.5.4.80:39524 - "POST /responses HTTP/1.1" 404 Not Found

from llama.cpp

llama-server[1658815]: srv  log_server_r: request: POST /responses 10.4.14.244 404

[iSugarSweet]

@ishaan-jaff In what way should this work? is there a setting available?

using 1.74.15, litellm still forwards /v1/responses to llama.cpp

litellm test key page using /v1/responses:

Error fetching response:Error: 404 litellm.NotFoundError: NotFoundError: OpenAIException - {"error":{"code":404,"message":"File Not Found","type":"not_found_error"}}. Received Model Group=

litellm logs:

INFO:     10.5.4.80:39524 - "POST /responses HTTP/1.1" 404 Not Found

from llama.cpp

llama-server[1658815]: srv  log_server_r: request: POST /responses 10.4.14.244 404

Did you have any luck getting this to work? Same issue for me.

Llama.cpp hosted model via llama-server, available through /v1/chat/completions.
Model registered in LiteLLM with calls to LiteLLM/v1/chat/completions working fine.

When attempting to call LiteLLM/v1/responses the 'Bridge' functionality does not appear to be working.

Model Config:

{
  "model_name": "openai/Qwen3-8B-Q4",
  "litellm_params": {
    "custom_llm_provider": "openai",
    "litellm_credential_name": "test",
    "use_in_pass_through": false,
    "use_litellm_proxy": false,
    "merge_reasoning_content_in_choices": false,
    "model": "openai/Qwen3-8B-Q4"
  },
  "model_info": {
    "id": "d0aa1dae-9721-4ae5-ae0f-d03b15930da5",
    "db_model": true,
    "access_via_team_ids": [
      "d8b9bbbd-fca7-45ab-a963-683eae4601bd"
    ],
    "direct_access": true
  },
  "provider": "openai",
  "litellm_model_name": "openai/Qwen3-8B-Q4",
  "cleanedLitellmParams": {
    "custom_llm_provider": "openai",
    "litellm_credential_name": "test",
    "use_in_pass_through": false,
    "use_litellm_proxy": false,
    "merge_reasoning_content_in_choices": false
  }
}

Logs Of Call:

litellm_proxy     | 10:22:31 - LiteLLM Proxy:ERROR: common_request_processing.py:642 - litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.NotFoundError: NotFoundError: OpenAIException - {"error":{"code":404,"message":"File Not Found","type":"not_found_error"}}. Received Model Group=openai/Qwen3-8B-Q4
litellm_proxy     | Available Model Group Fallbacks=None
litellm_proxy     | Traceback (most recent call last):
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 1668, in async_response_api_handler
litellm_proxy     |     response = await async_httpx_client.post(
litellm_proxy     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     ...<5 lines>...
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/logging_utils.py", line 135, in async_wrapper
litellm_proxy     |     result = await func(*args, **kwargs)
litellm_proxy     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/http_handler.py", line 324, in post
litellm_proxy     |     raise e
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/http_handler.py", line 280, in post
litellm_proxy     |     response.raise_for_status()
litellm_proxy     |     ~~~~~~~~~~~~~~~~~~~~~~~~~^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/httpx/_models.py", line 829, in raise_for_status
litellm_proxy     |     raise HTTPStatusError(message, request=request, response=self)
litellm_proxy     | httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://LLAMA.CPP_URL/v1/responses'
litellm_proxy     | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
litellm_proxy     |
litellm_proxy     | During handling of the above exception, another exception occurred:
litellm_proxy     |


litellm_proxy     | 10:22:31 - LiteLLM Proxy:ERROR: common_request_processing.py:642 - litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.NotFoundError: NotFoundError: OpenAIException - {"error":{"code":404,"message":"File Not Found","type":"not_found_error"}}. Received Model Group=openai/Qwen3-8B-Q4
litellm_proxy     | Available Model Group Fallbacks=None
litellm_proxy     | Traceback (most recent call last):
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 1668, in async_response_api_handler
litellm_proxy     |     response = await async_httpx_client.post(
litellm_proxy     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     ...<5 lines>...
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/logging_utils.py", line 135, in async_wrapper
litellm_proxy     |     result = await func(*args, **kwargs)
litellm_proxy     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/http_handler.py", line 324, in post
litellm_proxy     |     raise e
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/http_handler.py", line 280, in post
litellm_proxy     |     response.raise_for_status()
litellm_proxy     |     ~~~~~~~~~~~~~~~~~~~~~~~~~^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/httpx/_models.py", line 829, in raise_for_status
litellm_proxy     |     raise HTTPStatusError(message, request=request, response=self)
litellm_proxy     | httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://LLAMA.CPP_URL/v1/responses'
litellm_proxy     | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
litellm_proxy     |
litellm_proxy     | During handling of the above exception, another exception occurred:
litellm_proxy     |
litellm_proxy     | Traceback (most recent call last):
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/responses/main.py", line 309, in aresponses
litellm_proxy     |     response = await init_response
litellm_proxy     |                ^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 1677, in async_response_api_handler
litellm_proxy     |     raise self._handle_error(
litellm_proxy     | Traceback (most recent call last):
litellm_proxy     |         e=e,
litellm_proxy     |         ^^^^
litellm_proxy     |         provider_config=responses_api_provider_config,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 2409, in _handle_error
litellm_proxy     |     raise provider_config.get_error_class(
litellm_proxy     |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
litellm_proxy     |         error_message=error_text,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |         status_code=status_code,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |         headers=error_headers,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/base_llm/responses/transformation.py", line 200, in get_error_class
litellm_proxy     |     raise BaseLLMException(
litellm_proxy     |     ...<3 lines>...


litellm_proxy     | litellm.llms.base_llm.chat.transformation.BaseLLMException: {"error":{"code":404,"message":"File Not Found","type":"not_found_error"}}
litellm_proxy     |
litellm_proxy     | During handling of the above exception, another exception occurred:
litellm_proxy     |
litellm_proxy     | Traceback (most recent call last):
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/proxy/response_api_endpoints/endpoints.py", line 56, in responses_api
litellm_proxy     |     return await processor.base_process_llm_request(
litellm_proxy     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     ...<16 lines>...
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/proxy/common_request_processing.py", line 436, in base_process_llm_request
litellm_proxy     |     responses = await llm_responses
litellm_proxy     |                 ^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3576, in async_wrapper
litellm_proxy     |     return await self._ageneric_api_call_with_fallbacks(
litellm_proxy     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     ...<2 lines>...
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2711, in _ageneric_api_call_with_fallbacks
litellm_proxy     |     raise e
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2698, in _ageneric_api_call_with_fallbacks
litellm_proxy     |     response = await self.async_function_with_fallbacks(**kwargs)
litellm_proxy     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3902, in async_function_with_fallbacks
litellm_proxy     |     return await self.async_function_with_fallbacks_common_utils(
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/responses/main.py", line 309, in aresponses
litellm_proxy     |     )..<8 lines>...
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3860, in async_function_with_fallbacks_common_utils
litellm_proxy     |     raise original_exception
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3894, in async_function_with_fallbacks
litellm_proxy     |     response = await self.async_function_with_retries(*args, **kwargs)
litellm_proxy     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 4016, in async_function_with_retries
litellm_proxy     |     self.should_retry_this_error(
litellm_proxy     |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
litellm_proxy     |         error=e,
litellm_proxy     |         ^^^^^^^^
litellm_proxy     |     ...<4 lines>...
litellm_proxy     |         content_policy_fallbacks=content_policy_fallbacks,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 4190, in should_retry_this_error
litellm_proxy     |     raise error
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3990, in async_function_with_retries
litellm_proxy     |     response = await self.make_call(original_function, *args, **kwargs)
litellm_proxy     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 4108, in make_call
litellm_proxy     |     response = await response
litellm_proxy     |                ^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2790, in _ageneric_api_call_with_fallbacks_helper
litellm_proxy     |     raise e
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2776, in _ageneric_api_call_with_fallbacks_helper
litellm_proxy     |     response = await response  # type: ignore
litellm_proxy     |                ^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/utils.py", line 1586, in wrapper_async
litellm_proxy     |     raise e
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/utils.py", line 1437, in wrapper_async
litellm_proxy     |     result = await original_function(*args, **kwargs)
litellm_proxy     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/responses/main.py", line 326, in aresponses
litellm_proxy     |     raise litellm.exception_type(
litellm_proxy     |           ~~~~~~~~~~~~~~~~~~~~~~^
litellm_proxy     |         model=model,
litellm_proxy     |         ^^^^^^^^^^^^
litellm_proxy     |     ...<3 lines>...
litellm_proxy     |         extra_kwargs=kwargs,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2301, in exception_type
litellm_proxy     |     raise e
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 465, in exception_type
litellm_proxy     |     raise NotFoundError(
litellm_proxy     |     ...<5 lines>...
litellm_proxy     |     )
litellm_proxy     | litellm.exceptions.NotFoundError: litellm.NotFoundError: NotFoundError: OpenAIException - {"error":{"code":404,"message":"File Not Found","type":"not_found_error"}}. Received Model Group=openai/Qwen3-8B-Q4
litellm_proxy     | Available Model Group Fallbacks=None
litellm_proxy     | INFO:     10.130.164.246:8162 - "POST /responses HTTP/1.1" 404 Not Found
litellm_proxy     |     response = await init_response
litellm_proxy     |                ^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 1677, in async_response_api_handler
litellm_proxy     |     raise self._handle_error(
litellm_proxy     |           ~~~~~~~~~~~~~~~~~~^
litellm_proxy     |         e=e,
litellm_proxy     |         ^^^^
litellm_proxy     |         provider_config=responses_api_provider_config,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 2409, in _handle_error
litellm_proxy     |     raise provider_config.get_error_class(
litellm_proxy     |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
litellm_proxy     |         error_message=error_text,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |         status_code=status_code,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |         headers=error_headers,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/llms/base_llm/responses/transformation.py", line 200, in get_error_class
litellm_proxy     |     raise BaseLLMException(
litellm_proxy     |     ...<3 lines>...
litellm_proxy     |     )
litellm_proxy     | litellm.llms.base_llm.chat.transformation.BaseLLMException: {"error":{"code":404,"message":"File Not Found","type":"not_found_error"}}
litellm_proxy     |
litellm_proxy     | During handling of the above exception, another exception occurred:
litellm_proxy     |
litellm_proxy     | Traceback (most recent call last):
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/proxy/response_api_endpoints/endpoints.py", line 56, in responses_api
litellm_proxy     |     return await processor.base_process_llm_request(
litellm_proxy     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     ...<16 lines>...
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/proxy/common_request_processing.py", line 436, in base_process_llm_request
litellm_proxy     |     responses = await llm_responses
litellm_proxy     |                 ^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3576, in async_wrapper
litellm_proxy     |     return await self._ageneric_api_call_with_fallbacks(
litellm_proxy     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     ...<2 lines>...
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2711, in _ageneric_api_call_with_fallbacks
litellm_proxy     |     raise e
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2698, in _ageneric_api_call_with_fallbacks
litellm_proxy     |     response = await self.async_function_with_fallbacks(**kwargs)
litellm_proxy     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3902, in async_function_with_fallbacks
litellm_proxy     |     return await self.async_function_with_fallbacks_common_utils(
litellm_proxy     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     ...<8 lines>...
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3860, in async_function_with_fallbacks_common_utils
litellm_proxy     |     raise original_exception
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3894, in async_function_with_fallbacks
litellm_proxy     |     response = await self.async_function_with_retries(*args, **kwargs)
litellm_proxy     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 4016, in async_function_with_retries
litellm_proxy     |     self.should_retry_this_error(
litellm_proxy     |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
litellm_proxy     |         error=e,
litellm_proxy     |         ^^^^^^^^
litellm_proxy     |     ...<4 lines>...
litellm_proxy     |         content_policy_fallbacks=content_policy_fallbacks,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 4190, in should_retry_this_error
litellm_proxy     |     raise error
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 3990, in async_function_with_retries
litellm_proxy     |     response = await self.make_call(original_function, *args, **kwargs)
litellm_proxy     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 4108, in make_call
litellm_proxy     |     response = await response
litellm_proxy     |                ^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2790, in _ageneric_api_call_with_fallbacks_helper
litellm_proxy     |     raise e
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/router.py", line 2776, in _ageneric_api_call_with_fallbacks_helper
litellm_proxy     |     response = await response  # type: ignore
litellm_proxy     |                ^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/utils.py", line 1586, in wrapper_async
litellm_proxy     |     raise e
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/utils.py", line 1437, in wrapper_async
litellm_proxy     |     result = await original_function(*args, **kwargs)
litellm_proxy     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/responses/main.py", line 326, in aresponses
litellm_proxy     |     raise litellm.exception_type(
litellm_proxy     |           ~~~~~~~~~~~~~~~~~~~~~~^
litellm_proxy     |         model=model,
litellm_proxy     |         ^^^^^^^^^^^^
litellm_proxy     |     ...<3 lines>...
litellm_proxy     |         extra_kwargs=kwargs,
litellm_proxy     |         ^^^^^^^^^^^^^^^^^^^^
litellm_proxy     |     )
litellm_proxy     |     ^
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2301, in exception_type
litellm_proxy     |     raise e
litellm_proxy     |   File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 465, in exception_type
litellm_proxy     |     raise NotFoundError(
litellm_proxy     |     ...<5 lines>...
litellm_proxy     |     )
litellm_proxy     | litellm.exceptions.NotFoundError: litellm.NotFoundError: NotFoundError: OpenAIException - {"error":{"code":404,"message":"File Not Found","type":"not_found_error"}}. Received Model Group=openai/Qwen3-8B-Q4
litellm_proxy     | Available Model Group Fallbacks=None
litellm_proxy     | INFO:     10.130.164.246:8162 - "POST /responses HTTP/1.1" 404 Not Found

litellm_proxy     | INFO:     10.130.164.3:22114 - "GET / HTTP/1.1" 200 OK

[Mushoz]

@RuthgerD this should already work - please try it

I can confirm the findings of the other people in this issue that this indeed does not work. Are we missing a setting since you mentioned this is supposed to already work?

[neuhaus]

@ishaan-jaff can you please elaborate?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[kersta-staden]

I have the same issue.
My config is

model_list:
  - model_name: qwen3test
    litellm_params:
      model: openai/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8
      api_base: https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1
      api_key: "os.environ/STACKIT_API_KEY"
      mode: chat

and on ResponsesAPI Request I get the following error:

response = client.responses.create(
    model="qwen3test",
    input=[
    {
        "role": "user",
        "content": "Hi there"
    },
]
)

print(response)

NotFoundError: Error code: 404 - {'error': {'message': 'litellm.NotFoundError: NotFoundError: OpenAIException - Not found. Received Model Group=qwen3test\nAvailable Model Group Fallbacks=None', 'type': None, 'param': None, 'code': '404'}}

[tsic404]

I have the same issue. My config is

model_list:
  - model_name: qwen3test
    litellm_params:
      model: openai/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8
      api_base: https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1
      api_key: "os.environ/STACKIT_API_KEY"
      mode: chat

and on ResponsesAPI Request I get the following error:

response = client.responses.create(
    model="qwen3test",
    input=[
    {
        "role": "user",
        "content": "Hi there"
    },
]
)

print(response)

NotFoundError: Error code: 404 - {'error': {'message': 'litellm.NotFoundError: NotFoundError: OpenAIException - Not found. Received Model Group=qwen3test\nAvailable Model Group Fallbacks=None', 'type': None, 'param': None, 'code': '404'}}

try custom_openai/ instead of openai/