[Feature Request] Backoff more on resource exhausted error

[yiminc]

Is your feature request related to a problem? Please describe.

SDK has built in retry policy, which works pretty well for transient errors. Usually, retry will recover from those transient errors with only small number of retry attempt.
There is a list of retry-able errors, including ResourceExhaustedError. However, we should backoff much more on resource exhausted error because it is a clear indication that server is not able to keep up and aggressive retry will worsen the situation and could lead to more serious outage.

Describe the solution you'd like

Whenever seeing resource exhausted error, we need to backoff further. We may need a different retry policy with larger backoff interval and smaller max attempt count.

Additional context

Checklist:

• TypeScript
• Core (includes Python client)
• Go
• Java
• PHP

[cretz]

Any specific ideas on implementation here? Specifically:

What numbers do you propose? Right now in Go default gRPC retry is initial interval of 200ms w/ backoff exponent as 2 and max as 60s (or ctx deadline).

We may need a different retry policy with larger backoff interval and smaller max attempt count.

Can you clarify the logic here when, after resource exhausted retry policy is set, what we do when we receive non-resource-exhausted error? Should we completely reset the retry policy? Same question if it's a non-resource-exhausted error and then a resource exhausted error.

[yiminc]

Here is an example of how it can be handled just to give some idea:
https://github.com/temporalio/temporal/blob/3182dc2cde75e3e49fccb1de2905db5d30db58ab/common/backoff/retry.go#L204-L206

[bergundy]

@yiminc it would be ideal if the server let the SDK know what the base retry interval is for a given error so we don't have to update SDKs when the policy changes.

For now we will add the custom behavior as requested.
Not sure we'd want to do what the server retry does, I'd expect that attempt is persisted and the backoff duration is calculated based on the error type and the current attempt. In the server retrier, if the error flakes from "resource exhausted" to another error the attempt number is effectively reset.
So we'd replace the NextBackOff() method with a Backoff(attempt) method.