API should honor "Retry-After" header

[nddipiazza]

API should honor "Retry-After" header

Right now it just does a bland backoff with timed wait.

[mattwiller]

Hey @nddipiazza — in error scenarios (e.g. a 429 Too Many Requests or 500 Server Error response from the API), it's actually considered best practice to implement exponential backoff with randomized jitter — both Amazon and Google recommend this for their API consumers. This is due to the fact that in these types of situations, the API may not really be able to tell how long a given client should wait before making another request. Exponentially backing off ensures that client quickly falls below the rate limits and/or stops hammering API servers that may be failing due to high load. Randomized jitter helps prevent a thundering herd problem when multiple processes all retry in sync and generate excessive load on the API, by spreading those retries out over a window of time.

The Java SDK doesn't yet implement the randomized jitter, but it's something that's on our roadmap for the next couple months. Since we're planning to stick with the current strategy of exponential backoff for error scenarios, I'm going to close this issue on the related pull request. That said, please feel free to re-open for discussion if you'd like to talk further about it!

[nddipiazza]

What are your customers doing that are making millions of requests are they doing back off only or are they using the headers and doing back off? i.e. should i ignore the Retry-After parameter? Or still use it but also insert a random exponential backoff?

[mattwiller]

@nddipiazza For error scenarios (e.g. 429, 500) the recommendation is to just ignore the header and do exponential backoff with random jitter. For "success" responses that include a Retry-After (e.g. a 202), the header should be respected.

[nddipiazza]

Will do thank ye

…

On Jan 24, 2018 1:59 PM, "Matt Willer" ***@***.***> wrote: @nddipiazza <https://github.com/nddipiazza> For error scenarios (e.g. 429, 500) the recommendation is to just ignore the header and do exponential backoff with random jitter. For "success" responses that include a Retry-After (e.g. a 202), the header should be respected. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#438 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACIpOcO0JSg8SGV6Vk9oznyNRiamY2Hfks5tN4t3gaJpZM4Nm3wH> .

[nddipiazza]

hey @mattwiller this poses an issue for us.

right now box-java-sdk in master automatically attempts retries with no respect to expo retry nor retry-after header.

and without this PR large numbers of concurrent requests freezes completely (due to thundering herd problem).

When are you going to implement the expo retries? I'd really like to get off of custom versions of box-java-sdk in production.

If this isn't on schedule for many months... please tell me what retry library you are going to use? or are you going to just write it by hand with a while loop? Because I'll end up having to write the expo retry in a PR and don't want my code to get discarded when you eventually got to master with something different.

Being that the use case is pretty simple, adding a library like https://github.com/rhuffman/re-retrying is probably not going to be too necessary.

Please let me know.

[mattwiller]

Hey @nddipiazza, my understanding is that the SDK already implements exponential backoff in BackoffCounter — the calculateDelay() method does 2 << exponent, which is equivalent to Math.pow(2, exponent + 1). Are you seeing different behavior?

In any case, we're planning to ensure that the Java SDK has correct exponential backoff with randomized jitter in the next three months — probably by the end of March. We'll likely just implement it ourselves rather than using a library.

[nddipiazza]

@mattwiller yes. at one of your biggest clients we did extremely large file fetches and until we added the "Retry-After" waits, the fetches would completely freeze and not continue.

So whatever was there as of about 8 months ago definitely isn't sufficient.

Can you notify me when you add the jitter to the codebase? I can help you test it. I'll keep my ears open for 3 month-ish from now

[mattwiller]

@nddipiazza To help us make sure we're doing the right thing here, can you give us a bit more context on what you saw with those large file fetches?

1. What SDK method were you calling that froze? What does "froze" mean in this context?
2. How large is extremely large? More than 2GB?
3. Do you know what value you were using for api.getMaxRequestAttempts()?
4. Do you have any record or memory of what sort of values you saw coming back in the Retry-After header?
5. How did you implement the Retry-After wait? Just re-calling the SDK method after some period of time, or something more sophisticated?

In any case, I'll definitely make sure to keep you in the loop as we develop this functionality — any help testing in real-world scenarios would be very much appreciated.

[nddipiazza]

I was using "crawling" through files using the jwt service account. and i was fetching 1,000,000's of docs (TB's and TB's of content)

When increasing our thread count to any more than a few, the crawl would freeze.

We were simply doing a listing: list all files at users root, download each file recursively until out of files, move to the next one.

we did this in a "phased" approach where phase 1 we download all file metadata first, and phase 2 we download all the content.

We couldn't even list all the files and their metadata without it freezing completely where the API was simply always getting responses with "retry-after"

i don't remember what max atteps was.

The PR that was closed has the code i was using to get fixed.

[nddipiazza]

@mattwiller hi. how is the randomized jitter fix going? Is it still on track for march release?

[mattwiller]

@nddipiazza We added 50% randomization around the exponentially-increasing backoff time to try and avoid thundering herd problems, and also increased the default number of of retries from 3 to 5. That should help most clients work through getting hit with rate limits, since in the final retry they would be waiting anywhere from 16-48 seconds before making the API call again. In all but the most extreme cases, this should suffice to get back under the rate limit.

I appreciate the reminder — we actually released the new retry logic in v2.14.1 so it should be ready if you want to give it a try!