Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Azure client requests stuck issues on http.StatusTooManyRequests #81279

Merged
merged 1 commit into from Aug 13, 2019

Conversation

@feiskyer
Copy link
Member

commented Aug 12, 2019

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

/kind bug

What this PR does / why we need it:

Refer Azure/go-autorest#398. In go-autorest SDK https://github.com/Azure/go-autorest/blob/master/autorest/sender.go#L258,
if ARM returns http.StatusTooManyRequests, the sender doesn't increase the retry attempt count,
hence the Azure clients will keep retrying forever until it gets a status code other than 429.

// don't count a 429 against the number of attempts
// so that we continue to retry until it succeeds
if resp == nil || resp.StatusCode != http.StatusTooManyRequests {
    attempt++
}

So this PR explicitly remove http.StatusTooManyRequests from autorest.StatusCodesForRetry.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

  • Need cherry pick to v1.13-v1.15.

Does this PR introduce a user-facing change?:

Fix Azure client requests stuck issues on http.StatusTooManyRequests (HTTP Code 429).

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


/kind bug
/area provider/azure
/priority critial-urgent

@feiskyer

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

/sig cloud-provider
/assign @andyzhangx

@feiskyer

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

/priority critical-urgent

Copy link
Member

left a comment

the PR title could be Stop retry when got StatusTooManyRequests error ?

@feiskyer feiskyer changed the title Fix precondition cancel Fix Azure client requests stuck issues on http.StatusTooManyRequests Aug 12, 2019
Copy link
Member

left a comment

also this PR contains two fixes, would you split? and do you want to cherry pick?

@feiskyer

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

also this PR contains two fixes, would you split? and do you want to cherry pick?

Yep, it would be cherry-picked to old versions. Let me split the PR into two, cause they would be cherry picked to different versions.

@feiskyer feiskyer force-pushed the feiskyer:fix-precondition-cancel branch from 3d2b157 to 0bcee75 Aug 12, 2019
@feiskyer feiskyer force-pushed the feiskyer:fix-precondition-cancel branch from 0bcee75 to e53c57a Aug 12, 2019
@k8s-ci-robot k8s-ci-robot added size/S and removed size/M labels Aug 12, 2019
@feiskyer

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

/test pull-kubernetes-e2e-aks-engine-azure

Copy link
Member

left a comment

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Aug 13, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx, feiskyer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fejta-bot

This comment has been minimized.

Copy link

commented Aug 13, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 54b28ab into kubernetes:master Aug 13, 2019
24 checks passed
24 checks passed
cla/linuxfoundation feiskyer authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-aks-engine-azure Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-iscsi Skipped.
pull-kubernetes-e2e-gce-iscsi-serial Skipped.
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details
Provider Azure automation moved this from In progress to Done Aug 13, 2019
@k8s-ci-robot k8s-ci-robot added this to the v1.16 milestone Aug 13, 2019
k8s-ci-robot added a commit that referenced this pull request Aug 16, 2019
…79-upstream-release-1.14

Automated cherry pick of #81279: Fix Azure client requests stuck issues on
k8s-ci-robot added a commit that referenced this pull request Aug 16, 2019
…79-upstream-release-1.13

Automated cherry pick of #81279: Fix Azure client requests stuck issues on
k8s-ci-robot added a commit that referenced this pull request Aug 16, 2019
…79-upstream-release-1.15

Automated cherry pick of #81279: Fix Azure client requests stuck issues on
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.