Remove generic mount validation logic during unmount and rely on CSI driver #72008

oleksiys · 2018-12-13T01:58:12Z

What type of PR is this?
/kind bug

What this PR does / why we need it:
We remove a piece of generic logic which may not work with all the drivers and instead rely on CSI driver to run the teardown logic and report an error.

I faced an issue with a fuse mounted volume. After the fuse process responsible for the mount died, every time k8s or application tried to access the mounted volume it got an error "transport endpoint is not connected". I couldn't find a way to recover the fuse mount after that, the only thing I could do was unmount the volume. But once this happens you cannot release the volume anymore, because CSI driver which knows how to unmount the volume is never called because the check inside the TearDownAt function always fails.
I propose to remove that logic and rely on CSI driver's unmount logic.

/sig storage

k8s-ci-robot · 2018-12-13T01:58:13Z

@oleksiys: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2018-12-13T01:58:15Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: helpdesk@rt.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2018-12-13T01:58:20Z

Hi @oleksiys. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

oleksiys · 2018-12-13T02:08:50Z

The change was introduced in this PR #56836

oleksiys · 2018-12-13T07:22:33Z

I signed it
/check-cla

oleksiys · 2018-12-13T17:44:21Z

/assign @saad-ali

gnufied · 2018-12-13T21:20:42Z

pkg/volume/csi/csi_mounter.go

-	// see https://github.com/kubernetes/kubernetes/pull/56836#discussion_r155834524
-	mounted, err := isDirMounted(c.plugin, dir)
-	if err != nil {
-		klog.Error(log("unmounter.Teardown failed while checking mount status for dir [%s]: %v", dir, err))


How about rather than completely removing this check, if directory does not exist at all we do a fast return without calling CSI but if it does exist(mounted or not), we make the CSI call. This is what we do with flexvolume and it appears to have worked well in practice. cc @vladimirvivien

I wonder if this way we won't violate a lifecycle contract that for NodePublishVolume we must call NodeUnpublishVolume (at least once) so CSI driver have a chance to release resources even in situation when the actual mount point was deleted.

gnufied · 2018-12-13T21:21:21Z

/ok-to-test

saad-ali · 2018-12-20T01:46:54Z

This addresses #56836 (comment)

/lgtm
/approve

We should cherry pick this to 1.13.

/hold

Will let @gnufied @vladimirvivien review this. If they are ok with it, they can remove the hold.

k8s-ci-robot · 2018-12-20T01:47:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: oleksiys, saad-ali

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/volume/csi/OWNERS~~ [saad-ali]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

saad-ali · 2018-12-20T21:40:37Z

Since you are touching this code, could you also fix #72252 in this PR.

zhangxiaoyu-zidif · 2018-12-24T06:24:26Z

@oleksiys cloud you delete https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_mounter.go#L381 too in your commit?

pkg/volume/csi/csi_mounter.go

oleksiys · 2019-01-03T12:44:46Z

@gnufied @saad-ali PTAL

gnufied · 2019-01-05T03:00:14Z

/hold cancel
/lgtm

gnufied · 2019-01-05T03:00:54Z

err, can you squash the commits please? since both commits are made to the same file, it seems pointless.

/lgtm cancel

gnufied · 2019-01-07T17:54:42Z

/release-note-none
/lgtm

…2008-upstream-release-1.13 Automated cherry pick of #72008: Fix CSI volume unmount and cleanup logic

Remove generic mount validation logic during unmount and rely on CSI driver ref: kubernetes#72008 See merge request !206664

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 13, 2018

k8s-ci-robot requested review from gnufied and msau42 December 13, 2018 01:58

oleksiys force-pushed the master branch from 2d0d1b7 to 0741639 Compare December 13, 2018 06:00

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Dec 13, 2018

k8s-ci-robot assigned saad-ali Dec 13, 2018

gnufied reviewed Dec 13, 2018

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 13, 2018

This was referenced Dec 19, 2018

revert mount check because csi-driver won't unmount as expected #72194

Closed

CSI bug fixes for teardown, nodepublish probe, target path creation #56836

Merged

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Dec 20, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 20, 2018

saad-ali mentioned this pull request Dec 20, 2018

CSI volume directory is not completely cleaned up after volume is unmounted #72252

Closed

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 2, 2019

msau42 reviewed Jan 3, 2019

View reviewed changes

pkg/volume/csi/csi_mounter.go Show resolved Hide resolved

oleksiys force-pushed the master branch 2 times, most recently from b95ca6c to acabed4 Compare January 3, 2019 00:38

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 3, 2019

oleksiys force-pushed the master branch from acabed4 to e7d569f Compare January 3, 2019 02:57

k8s-ci-robot assigned gnufied Jan 5, 2019

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jan 5, 2019

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 5, 2019

Fix CSI volume unmount and cleanup logic

f6a0359

oleksiys force-pushed the master branch from e7d569f to f6a0359 Compare January 5, 2019 17:04

k8s-ci-robot merged commit ee10b45 into kubernetes:master Jan 7, 2019

saad-ali mentioned this pull request Jan 30, 2019

CSI plugin registration problem after kublet restarts #72500

Closed

chrischdi mentioned this pull request Jun 3, 2019

kubelet/csi fails to unmount on IO errors #77969

Closed

This was referenced Sep 30, 2019

k8s 1.13 Cherry-pick: Fix CSI volume unmount and cleanup logic #83334

Closed

Automated cherry pick of #72008: Fix CSI volume unmount and cleanup logic #83338

Merged

k8s-ci-robot added a commit that referenced this pull request Oct 2, 2019

Merge pull request #83338 from ggriffiths/automated-cherry-pick-of-#7…

818826a

…2008-upstream-release-1.13 Automated cherry pick of #72008: Fix CSI volume unmount and cleanup logic

mars1024 pushed a commit to mars1024/kubernetes that referenced this pull request Nov 26, 2019

Merge branch 'delete-csi-mounted-check' into 'develop'

5051fde

Remove generic mount validation logic during unmount and rely on CSI driver ref: kubernetes#72008 See merge request !206664

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove generic mount validation logic during unmount and rely on CSI driver #72008

Remove generic mount validation logic during unmount and rely on CSI driver #72008

oleksiys commented Dec 13, 2018

k8s-ci-robot commented Dec 13, 2018

k8s-ci-robot commented Dec 13, 2018

k8s-ci-robot commented Dec 13, 2018

oleksiys commented Dec 13, 2018

oleksiys commented Dec 13, 2018

oleksiys commented Dec 13, 2018

gnufied Dec 13, 2018

oleksiys Dec 14, 2018

gnufied commented Dec 13, 2018

saad-ali commented Dec 20, 2018

k8s-ci-robot commented Dec 20, 2018

saad-ali commented Dec 20, 2018

zhangxiaoyu-zidif commented Dec 24, 2018

oleksiys commented Jan 3, 2019

gnufied commented Jan 5, 2019

gnufied commented Jan 5, 2019

gnufied commented Jan 7, 2019

Remove generic mount validation logic during unmount and rely on CSI driver #72008

Remove generic mount validation logic during unmount and rely on CSI driver #72008

Conversation

oleksiys commented Dec 13, 2018

k8s-ci-robot commented Dec 13, 2018

k8s-ci-robot commented Dec 13, 2018

k8s-ci-robot commented Dec 13, 2018

oleksiys commented Dec 13, 2018

oleksiys commented Dec 13, 2018

oleksiys commented Dec 13, 2018

gnufied Dec 13, 2018

Choose a reason for hiding this comment

oleksiys Dec 14, 2018

Choose a reason for hiding this comment

gnufied commented Dec 13, 2018

saad-ali commented Dec 20, 2018

k8s-ci-robot commented Dec 20, 2018

saad-ali commented Dec 20, 2018

zhangxiaoyu-zidif commented Dec 24, 2018

oleksiys commented Jan 3, 2019

gnufied commented Jan 5, 2019

gnufied commented Jan 5, 2019

gnufied commented Jan 7, 2019