[Bug]: Model Overloaded exception

[andrewn3]

What happened?

Would be good if you handle this scenario and pause and retry a few minutes later rather than throwing an exception as in this case the application will bomb out.

Also the api key in the exception message should really be obfuscated but i manually removed it.

Relevant log output

15:36:34 - LiteLLM:WARNING: utils.py:428 - `litellm.set_verbose` is deprecated. Please set `os.environ['LITELLM_LOG'] = 'DEBUG'` for debug logs.
WARNING:LiteLLM:`litellm.set_verbose` is deprecated. Please set `os.environ['LITELLM_LOG'] = 'DEBUG'` for debug logs.
Initialized litellm callbacks, Async Success Callbacks: [<litellm.integrations.langfuse.langfuse_prompt_management.LangfusePromptManagement object at 0x000002CD95369730>, 'cache', <crewai.utilities.token_counter_callback.TokenCalcHandler object at 0x000002CD99E37B90>]
SYNC kwargs[caching]: False; litellm.cache: None; kwargs.get('cache')['no-cache']: False
Final returned optional params: {'stop_sequences': ['\nObservation:']}


LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.


Provider List: https://docs.litellm.ai/docs/providers

ERROR:root:LiteLLM call failed: litellm.InternalServerError: litellm.InternalServerError: VertexAIException - {
  "error": {
    "code": 503,
    "message": "The model is overloaded. Please try again later.",
    "status": "UNAVAILABLE"
  }
}

 Error during LLM call: litellm.InternalServerError: litellm.InternalServerError: VertexAIException - {
  "error": {
    "code": 503,
    "message": "The model is overloaded. Please try again later.",
    "status": "UNAVAILABLE"
  }
}

[Flow._execute_single_listener] Error in method PDFExtractor: litellm.InternalServerError: litellm.InternalServerError: VertexAIException - {
  "error": {
    "code": 503,
    "message": "The model is overloaded. Please try again later.",
    "status": "UNAVAILABLE"
  }
}

Traceback (most recent call last):
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\litellm\llms\vertex_ai\gemini\vertex_and_google_ai_studio_gemini.py", line 1282, in completion
    response = client.post(url=url, headers=headers, json=data)  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\litellm\llms\custom_httpx\http_handler.py", line 553, in post
    raise e
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\litellm\llms\custom_httpx\http_handler.py", line 534, in post
    response.raise_for_status()
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\httpx\_models.py", line 763, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '503 Service Unavailable' for url 'https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=<manually removed>'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\litellm\main.py", line 2299, in completion
    response = vertex_chat_completion.completion(  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\litellm\llms\vertex_ai\gemini\vertex_and_google_ai_studio_gemini.py", line 1286, in completion
    raise VertexAIError(
litellm.llms.vertex_ai.common_utils.VertexAIError: {
  "error": {
    "code": 503,
    "message": "The model is overloaded. Please try again later.",
    "status": "UNAVAILABLE"
  }
}


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\flow\flow.py", line 990, in _execute_single_listener
    listener_result = await self._execute_method(listener_name, method)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\flow\flow.py", line 825, in _execute_method
    else method(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\src\productmanagementflow\main.py", line 120, in PDFExtractor
    .kickoff(inputs={"pdf_filename": self.state.pdffilename,
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\crew.py", line 578, in kickoff
    result = self._run_hierarchical_process()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\crew.py", line 688, in _run_hierarchical_process
    return self._execute_tasks(self.tasks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\crew.py", line 781, in _execute_tasks
    task_output = task.execute_sync(
                  ^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\task.py", line 302, in execute_sync
    return self._execute_core(agent, context, tools)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\task.py", line 366, in _execute_core
    result = agent.execute_task(
             ^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\agent.py", line 254, in execute_task
    raise e
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\agent.py", line 243, in execute_task
    result = self.agent_executor.invoke(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\agents\crew_agent_executor.py", line 112, in invoke
    raise e
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\agents\crew_agent_executor.py", line 102, in invoke
    formatted_answer = self._invoke_loop()
                       ^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\agents\crew_agent_executor.py", line 160, in _invoke_loop
    raise e
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\agents\crew_agent_executor.py", line 140, in _invoke_loop
    answer = self._get_llm_response()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\agents\crew_agent_executor.py", line 210, in _get_llm_response
    raise e
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\agents\crew_agent_executor.py", line 201, in _get_llm_response
    answer = self.llm.call(
             ^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\crewai\llm.py", line 291, in call
    response = litellm.completion(**params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\litellm\utils.py", line 1154, in wrapper
    raise e
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\litellm\utils.py", line 1032, in wrapper
    result = original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\litellm\main.py", line 3068, in completion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\litellm\litellm_core_utils\exception_mapping_utils.py", line 2201, in exception_type
    raise e
  File "D:\Users\newla\crewai\financial_analyst1\productmanager\productmanagementflow\.venv\Lib\site-packages\litellm\litellm_core_utils\exception_mapping_utils.py", line 1197, in exception_type
    raise litellm.InternalServerError(
litellm.exceptions.InternalServerError: litellm.InternalServerError: litellm.InternalServerError: VertexAIException - {
  "error": {
    "code": 503,
    "message": "The model is overloaded. Please try again later.",
    "status": "UNAVAILABLE"
  }
}

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

1.60.2

Twitter / LinkedIn details

No response

[colesmcintosh]

@andrewn3 working on a fix for this

[colesmcintosh]

@andrewn3 there is built-in support for retries and fallbacks to manage such transient errors, here are the docs --> https://docs.litellm.ai/docs/proxy/reliability

I'll open a PR to cover the api key masking 👍

[colesmcintosh]

@andrewn3 opened this PR for api key masking #9285

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.