Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically remove orphaned pod's dangling volumes #95301

Merged
merged 5 commits into from Feb 16, 2021
Merged

Automatically remove orphaned pod's dangling volumes #95301

merged 5 commits into from Feb 16, 2021

Conversation

lorenz
Copy link
Contributor

@lorenz lorenz commented Oct 5, 2020

What type of PR is this?

/kind bug

What this PR does / why we need it:
This makes Kubelet automatically clean up dangling volumes for orphaned pods and also emits warnings when it does so. This fixes the endless log spam about orphaned volumes (Orphaned pod <xyz> found, but volume paths are still present on disk) and replaces it with a single useful message (Removed dangling volume directories for orphaned pod <xyz> or Removed dangling subpath directories for orphaned pod <xyz>). This still allows for debugging issues with volume reconstruction without annoying operators excessively. People are currently using highly unsafe scripts to work around this issue (like #60987 (comment)).

As of Kubernetes 1.19 volume reconstruction still sometimes fails just with emptyDirs and secrets and no kubelet restarts.

This is a rewrite of #73799 which also deals with subpaths and uses more K8s utility code.

Which issue(s) this PR fixes:
Fixes #60987 (partially)

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Kubelet now cleans up orphaned volume directories automatically

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 5, 2020
@k8s-ci-robot
Copy link
Contributor

Hi @lorenz. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 5, 2020
@lorenz
Copy link
Contributor Author

lorenz commented Oct 5, 2020

/sig storage

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 5, 2020
@k8s-ci-robot k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. labels Oct 5, 2020
@dims
Copy link
Member

dims commented Oct 22, 2020

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 22, 2020
@dims
Copy link
Member

dims commented Oct 22, 2020

/assign @msau42 @sjenning @derekwaynecarr

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 22, 2020
@jingxu97
Copy link
Contributor

/assign @jingxu97

@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 22, 2020
@lorenz
Copy link
Contributor Author

lorenz commented Oct 22, 2020

I rebased onto current master, but all K8s tests somehow fail with a merge conflict.

@k8s-ci-robot k8s-ci-robot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Feb 8, 2021
@jingxu97 jingxu97 added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Feb 8, 2021
@kikisdeliveryservice kikisdeliveryservice moved this from Needs Reviewer to Needs Approver in SIG Node PR Triage Feb 11, 2021
@bobbypage
Copy link
Member

/cc

@dchen1107
Copy link
Member

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dchen1107, jingxu97, lorenz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 16, 2021
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Feb 16, 2021 •
edited

@lorenz: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-e2e-gce-csi-serial 322202a52d4f427ab9b0df36aafb5a9bfa340061 link /test pull-kubernetes-e2e-gce-csi-serial
pull-kubernetes-e2e-gce-storage-slow 322202a52d4f427ab9b0df36aafb5a9bfa340061 link /test pull-kubernetes-e2e-gce-storage-slow
pull-kubernetes-e2e-azure-disk-windows 322202a52d4f427ab9b0df36aafb5a9bfa340061 link /test pull-kubernetes-e2e-azure-disk-windows
pull-kubernetes-e2e-gce-storage-snapshot 322202a52d4f427ab9b0df36aafb5a9bfa340061 link /test pull-kubernetes-e2e-gce-storage-snapshot
pull-kubernetes-e2e-azure-file-windows 322202a52d4f427ab9b0df36aafb5a9bfa340061 link /test pull-kubernetes-e2e-azure-file-windows

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lorenz
Copy link
Contributor Author

lorenz commented Feb 16, 2021

/retest

@k8s-ci-robot k8s-ci-robot merged commit 55626cf into kubernetes:master Feb 16, 2021
SIG Node PR Triage automation moved this from Needs Approver to Done Feb 16, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Feb 16, 2021
@lorenz lorenz deleted the pr-remove-dangling-volumes branch February 16, 2021 22:36
@bobbypage
Copy link
Member

bobbypage commented Feb 17, 2021 •
edited

Hi @lorenz, thanks for working on this. Do you think we should cherrypick this to older release branches i.e. say 1.19 and 1.20?

/cc @jingxu97

@jingxu97
Copy link
Contributor

I think we should cherrypick

@lorenz
Copy link
Contributor Author

lorenz commented Feb 18, 2021

@bobbypage Cherry-picking sounds reasonable as this is a fairly wide-spread issue.

@joshimoo joshimoo mentioned this pull request Feb 22, 2021
Closed
@bobbypage
Copy link
Member

thanks @lorenz. Do you want to propose cherrypick? If not or you're busy happy to help create cherrypick of this PR myself. Let me know :)

@amal-kumar
Copy link

Hi All,
We are facing the same issue on our GKE cluster. Please can you let us know what is the plan to release this fix?

@bobbypage bobbypage mentioned this pull request Mar 10, 2021
Merged
@bobbypage
Copy link
Member

Cherrypick for 1.20 created - #100060

@bobbypage bobbypage mentioned this pull request Mar 10, 2021
Merged
@bobbypage
Copy link
Member

Cherrypick for 1.19 created - #100104

@ehashman ehashman mentioned this pull request Mar 12, 2021
Merged
k8s-ci-robot added a commit that referenced this pull request Mar 12, 2021
@k8s-ci-robot
Merge pull request #100104 from bobbypage/automated-cherry-pick-of-#9…
67aa54c 
…5301-upstream-release-1.19

Automated cherry pick of #95301: Automatically remove orphaned pod's dangling volumes
k8s-ci-robot added a commit that referenced this pull request Mar 12, 2021
@k8s-ci-robot
Merge pull request #100190 from ehashman/automated-cherry-pick-of-#95…
1c2bfc6 
…301-upstream-release-1.18

Automated cherry pick of #95301: Automatically remove orphaned pod's dangling volumes
k8s-ci-robot added a commit that referenced this pull request Mar 13, 2021
@k8s-ci-robot
Merge pull request #100060 from bobbypage/automated-cherry-pick-of-#9…
4b39e92 
…5301-upstream-release-1.20

Automated cherry pick of #95301: Automatically remove orphaned pod's dangling volumes
@rdxmb rdxmb mentioned this pull request Mar 17, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jingxu97 jingxu97 jingxu97 left review comments

@bobbypage bobbypage Awaiting requested review from bobbypage

Assignees

@gnufied gnufied

@sjenning sjenning

@jsafrane jsafrane

@Random-Liu Random-Liu

@derekwaynecarr derekwaynecarr

@yujuhong yujuhong

@jingxu97 jingxu97

@msau42 msau42

@saikat-royc saikat-royc

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node PR Triage
Done
Milestone
v1.21
Development

Successfully merging this pull request may close these issues.

Orphaned pod found - but volume paths are still present on disk