Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Reduce Azure API calls by replacing the current backoff retry with SDK's backoff #70866
What type of PR is this?
What this PR does / why we need it:
When something unexpected happened (e.g. hitting Azure API throttling) , Azure cloud provider would retry with a configurable exponential backoff. While this seems work for processing various unexpected errors, it's actually sending too much requests to Azure API.
That means every failed
We should really avoid those two-layer backoff retries by
According to Throttling Resource Manager requests, when the request limits are reached, clients should honor Retry-After value in the response header:
Since this header has already been handled within SDK's backoff retries, (2) is our preferred way to reduce the API calls.
Which issue(s) this PR fixes (optional, in
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: feiskyer
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing
I'm a little bit worried about taking out the manual configuration knobs and deferring entirely to the SDK.
I think that we should both maintain the existing behavior, and implement the new one, and switch between them based on a flag.
Since this is a big change, we need to maintain backward/rollback as an option for clusters without reverting the Kubernetes version.
Reasonable. Since there are two layers of retrying today, we should provide a way to disable either SDK's retry or cloud provider's retry.
left a comment
Just one comment really. I think we are taking on a lot of tech debt maintaining two modes of
We can also emit a warning message as if we found that user is still supplying values for the old backoff style.