[Bug]: Sudden switch to responses from completions causing failures

[willkoh]

What happened?

PR #16766 was merged in at night, and models started using responses instead of completions. The bridge does not work automatically for many cases, and structured outputs also failed for these requests.

Can we have an env var that does not auto-bridge completions to responses? Currently, our workaround is setting LITELLM_LOCAL_MODEL_COST_MAP="True".

Relevant log output

litellm.ContextWindowExceededError: litellm.BadRequestError: ContextWindowExceededError: OpenAIException - {
  "error": {
    "message": "Invalid 'metadata.schema_dict_json': string too long. Expected a string with maximum length 512, but got a string with length 1203 instead.",
    "type": "invalid_request_error",
    "param": "metadata.schema_dict_json",
    "code": "string_above_max_length"
  }
}

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.80.0

Twitter / LinkedIn details

No response

[eugepemi]

What happened?

Related to this issue, we started to have failures in our side because this change too

Relevant log output

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 1668, in async_response_api_handler
    response = await async_httpx_client.post(
  File "/usr/local/lib/python3.10/site-packages/litellm/litellm_core_utils/logging_utils.py", line 135, in async_wrapper
    result = await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/custom_httpx/http_handler.py", line 324, in post
    raise e
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/custom_httpx/http_handler.py", line 280, in post
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.openai.com/v1/responses'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/litellm/responses/main.py", line 309, in aresponses
    response = await init_response
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 1677, in async_response_api_handler
    raise self._handle_error(
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 2409, in _handle_error
    raise provider_config.get_error_class(
  File "/usr/local/lib/python3.10/site-packages/litellm/llms/base_llm/responses/transformation.py", line 200, in get_error_class
    raise BaseLLMException(
litellm.llms.base_llm.chat.transformation.BaseLLMException: {
  "error": {
    "message": "Missing required parameter: 'tools[0].name'.",
    "type": "invalid_request_error",
    "param": "tools[0].name",
    "code": "missing_required_parameter"
  }
}

Are you a ML Ops Team?

More or less

What LiteLLM version are you on ?

v1.80.0

[adrienbrault]

@willkoh Thank you for the workaround!

I never expected litellm to behave differently without me explicitly updating the library.
This is quite problematic

[fcabissi]

How is it possible that my requests are being routed to a different endpoint without having changed the library version? I am seeing this in v1.79.1

[willkoh]

@fcabissi The model costs are fetched over the network from this json file which serves this file in the repo. Any changes to this file are picked up automatically due to how it's fetched, which makes sense normally that costs should dynamically change without having to update the package. However, tying a core package functionality into it wasn't expected on our end.

The relevant code path starts here.

Setting LITELLM_LOCAL_MODEL_COST_MAP="True" pulls the costs from the "backup" json file in the package, which only updates with package updates.

[guscl]

Can the litellm team just revert the PR?