[Bug] OpenAI agent sdk non retryable error on liteLLM 503

[morgan-cromell]

What are you really trying to do?

Using the openai agent sdk with temporal and LiteLLM with aws bedrock. For example the anthropic models.

Describe the bug

When you get rate limited by aws bedrock it throws an litellm.exceptions.ServiceUnavailableError

For example:
litellm.ServiceUnavailableError: BedrockException - {"message":"Model is getting throttled. Try your request again."}

This is a retryable error and should produce a retry of the activity. Instead it results in this error
temporalio.exceptions.ApplicationError: Non retryable OpenAI status code

Minimal Reproduction

Setup one of the openAI agent SDK temporal samples and use the LiteLLM provider. Use one of the models that has a lot of throttling and wait until you get throttled/rate limited.

Environment/Versions

• OS and processor: [e.g. M1 Mac, x86 Windows, Linux]
• Temporal Version: [e.g. 1.14.0?] and/or SDK version
• Are you using Docker or Kubernetes or building Temporal from source?
  Temporal python sdk 1.18.1

Additional context

[tconley1428]

We tried to mimic the retry policies implemented by OpenAI. https://github.com/openai/openai-python/blob/dff16b5e9964bf85157eb41181255ed5dde8dda4/src/openai/_base_client.py#L750. In this particular case, given that the error did not provide retry headers, we base it on the status code reported by the underlying provider. LiteLLM is turning it into https://github.com/BerriAI/litellm/blob/57a2ec3beb7fcfb7ec7ccd379a19d2db951da9ef/litellm/exceptions.py#L485 which has a status code of 503. We could theoretically change our retry behavior on that, but I'm wary of creating that discrepancy, since the same retry logic is used for litellm and openai.

It does seem like LiteLLM's throttling error detection is either incorrect or insufficient for bedrock: https://github.com/BerriAI/litellm/blob/57a2ec3beb7fcfb7ec7ccd379a19d2db951da9ef/litellm/litellm_core_utils/exception_mapping_utils.py#L942, which is something you could report to them. If it was converted to a 429, retry would happen.

I think the more likely thing for us to do would be to add an additional configuration point to let you do your own exception handling for your particular needs.

[morgan-cromell]

@tconley1428 your implementation does not correctly mimic the openAI retry policies. You only retry on 500 errors while the openAI code retries on >= 500
https://github.com/openai/openai-python/blob/dff16b5e9964bf85157eb41181255ed5dde8dda4/src/openai/_base_client.py#L778

        # Retry internal errors.
        if response.status_code >= 500:
            log.debug("Retrying due to status code %i", response.status_code)
            return True

[tconley1428]

Ah, thank you. I definitely missed the inequality comparison there. That's a straightforward fix then.