New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete pods assigned to not existing nodes. #73401

Merged
merged 1 commit into from Jan 30, 2019

Conversation

Projects
None yet
5 participants
@krzysztof-jastrzebski
Copy link
Contributor

krzysztof-jastrzebski commented Jan 28, 2019

What type of PR is this?
/kind bug

What this PR does / why we need it:
PR adds deleting pods created by DaemonSet assigned to not existing nodes.
Which issue(s) this PR fixes:
Fixes #71349

Does this PR introduce a user-facing change?:

Adds deleting pods created by DaemonSet assigned to not existing nodes.
@krzysztof-jastrzebski

This comment has been minimized.

Copy link
Contributor Author

krzysztof-jastrzebski commented Jan 28, 2019

/ok-to-test

@krzysztof-jastrzebski

This comment has been minimized.

Copy link
Contributor Author

krzysztof-jastrzebski commented Jan 28, 2019

/sig apps

@krzysztof-jastrzebski krzysztof-jastrzebski force-pushed the krzysztof-jastrzebski:daemonsets branch from de5bea5 to 7186d3b Jan 28, 2019

@krzysztof-jastrzebski

This comment has been minimized.

Copy link
Contributor Author

krzysztof-jastrzebski commented Jan 28, 2019

/test pull-kubernetes-node-e2e

@krzysztof-jastrzebski

This comment has been minimized.

Copy link
Contributor Author

krzysztof-jastrzebski commented Jan 28, 2019

/test pull-kubernetes-node-e2e
/test pull-kubernetes-e2e-gce

@krzysztof-jastrzebski

This comment has been minimized.

Copy link
Contributor Author

krzysztof-jastrzebski commented Jan 28, 2019

/test pull-kubernetes-e2e-gce
/test pull-kubernetes-node-e2e

@k82cn

This comment has been minimized.

Copy link
Member

k82cn commented Jan 29, 2019

/assign


// getPodsWithoutNode returns list of pods assigned to not existing nodes.
func getPodsWithoutNode(
runningNodesList []*v1.Node, nodeToDaemonPods map[string][]*v1.Pod) []string {

This comment has been minimized.

@k82cn

k82cn Jan 29, 2019

Member

move to a single line :)

@k82cn

This comment has been minimized.

Copy link
Member

k82cn commented Jan 29, 2019

LGTM ; let's get it merged when CI's happy :)

@k82cn

This comment has been minimized.

Copy link
Member

k82cn commented Jan 29, 2019

/retest

@k82cn

This comment has been minimized.

Copy link
Member

k82cn commented Jan 30, 2019

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Jan 30, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 30, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: k82cn, krzysztof-jastrzebski

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit f787705 into kubernetes:master Jan 30, 2019

18 checks passed

cla/linuxfoundation krzysztof-jastrzebski authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-godeps Skipped
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
@mikedanese

This comment has been minimized.

Copy link
Member

mikedanese commented Jan 30, 2019

isn't this smoothing that the node controller should handle in general? was the problem that they were critical pods?

@seh

This comment has been minimized.

Copy link
Contributor

seh commented Jan 30, 2019

These pods weren't assigned to nodes by the scheduler yet; they were created with affinity that will only allow them to be placed on a single node—in each case, a node that no longer exists.

This happens with pods regardless of whether they're critical or not. We see it happen with pods that need to wait until the node is ready, where the node never became ready before it got deleted.

@k82cn

This comment has been minimized.

Copy link
Member

k82cn commented Jan 31, 2019

isn't this smoothing that the node controller should handle in general? was the problem that they were critical pods?

PodGC will check pod's node name to clean them up. After ScheduleDaemonPod (schedule daemonset pods by scheduler), DemonSetController will only append node affinity with target node and let scheduler to do binding. So it's better to handle it in DaemonSetController.

BTW, @krzysztof-jastrzebski , would you help to add feature-gate checking there? ScheduleDaemonPod is still beta now; if anyone disable it, we don-t need to check all-nodes and all-pods :)

k8s-ci-robot added a commit that referenced this pull request Feb 12, 2019

Merge pull request #73710 from krzysztof-jastrzebski/automated-cherry…
…-pick-of-#73401-#73606-upstream-release-1.12

Automated cherry pick of #73401: Delete pods assigned to not existing nodes. #73606: Deleting pods assigned to not existing nodes only if
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment