Implement retries on 503s

[tianon]

This caps total attempts of any given request to at most 3x 503 responses (in other words, the third 503 will "bubble up" as-is). The intent there is that if there is a serious issue with Hub, we don't want to continue retrying aggressively as that would likely contribute to the outage persisting longer.

(inspired by the recurring random 503s we keep getting that resolve on a retry)

[tianon]

Hmmm, maybe we should also consider checking the Retry-After header (which is valid for both 503 and 429; https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After). 🤔

[tianon]

(I have no idea whether Hub is sending Retry-After with these though -- error reporting for these errors in almost all tools rarely includes the headers for a lot of good reasons 🙈)

[whalelines]

(inspired by the recurring random 503s we keep getting that resolve on a retry)

So we want to retry, but not too many times?

maybe we should also consider checking the Retry-After header

Probably worth checking, after this is merged.

[tianon]

Yeah, retrying one request is lighter weight than restarting the whole job, and should be enough for spurious 503s (and the explicit limit helps protect Hub if they aren't spurious 👍) 👀