Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ignore failed pods to not block rolling update daemonset #78170

Merged
merged 1 commit into from Jul 24, 2019

Conversation

@DaiHao
Copy link
Member

commented May 21, 2019

What type of PR is this?

/kind bug

What this PR does / why we need it:

daemon set rolling update hang when there exist failed pods on node
As long as there exists one failed pod of daemonset, it will block the entire upgrade process.

Which issue(s) this PR fixes:

Fixes #63465

Special notes for your reviewer:

#65309 has used backoff algorithm to avoid hot loops fighting with kubelet on pod recreation, so it's safe to return nil in 'manage' method when there exist failed pod.
If not, 'manage' method returns error which triggers syncDaemonSet method returns directly. After failed pod be deleted, a new pod will be created but may still in failed state if kubelet reject it. So we got a another failed pod and fall into an infinite loop.

/assign @janetkuo @k82cn
/sig apps
/kind bug
/priority important-soon

Does this PR introduce a user-facing change?:

Fix a bug that causes DaemonSet rolling update hang when there exist failed pods. 
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented May 21, 2019

Hi @DaiHao. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot requested review from mortent and smarterclayton May 21, 2019

@DaiHao DaiHao changed the title ignore failed pods to not stuck rolling update daemonset ignore failed pods to not block rolling update daemonset May 21, 2019

@k82cn

This comment has been minimized.

Copy link
Member

commented May 21, 2019

/assign

I'd like to go through this firstly to see what happened there :)

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented May 27, 2019

@DaiHao: You must be a member of the kubernetes/kubernetes-milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@DaiHao

This comment has been minimized.

Copy link
Member Author

commented May 27, 2019

/assign

I'd like to go through this firstly to see what happened there :)

ptal, it's supposed to be merged in v1.15.

@zhangxiaoyu-zidif

This comment has been minimized.

Copy link
Member

commented May 28, 2019

/ok-to-test

@DaiHao

This comment has been minimized.

Copy link
Member Author

commented May 28, 2019

/test pull-kubernetes-kubemark-e2e-gce-big

@DaiHao

This comment has been minimized.

Copy link
Member Author

commented Jun 14, 2019

@k82cn @janetkuo this pr has pending for a while, ptal

@DaiHao

This comment has been minimized.

Copy link
Member Author

commented Jul 11, 2019

ping @k82cn

@ScorpioCPH

This comment has been minimized.

Copy link
Member

commented Jul 12, 2019

/lgtm

@janetkuo

This comment has been minimized.

Copy link
Member

commented Jul 24, 2019

/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jul 24, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DaiHao, janetkuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@DaiHao

This comment has been minimized.

Copy link
Member Author

commented Jul 24, 2019

/test pull-kubernetes-kubemark-e2e-gce-big

@k8s-ci-robot k8s-ci-robot merged commit bbdd055 into kubernetes:master Jul 24, 2019

20 checks passed

cla/linuxfoundation DaiHao authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.