Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip mount point checks when possible during mount cleanup. #109676

Merged
merged 8 commits into from Jul 14, 2022

Conversation

cartermckinnon
Copy link

@cartermckinnon cartermckinnon commented Apr 26, 2022

What type of PR is this?

/kind bug
/kind api-change

What this PR does / why we need it:

This is a continuation of #109117. Please see that PR for more background.

Calls to mounter.Unmount are preceded and followed by expensive mount point checks. These checks are not necessary on *nix-s with a umount implementation that performs a similar check itself. This PR adds a mechanism to detect the "safe" behavior, and avoid mount point checks when possible.

This change represents a significant optimization of CleanupMountPoint; enabling use-cases where pods have many mounts, and there is high pod churn. We (EKS) have observed several cases of instability and poor node health in such scenarios, which were resolved with this change.

Which issue(s) this PR fixes:

No issue available.

Special notes for your reviewer:

I chose to add a function to the Mounter interface in order to keep the CleanupMountPoint implementation generic for Unix and Windows. If the "safe" umount behavior is not detected, the existing code paths are unchanged.

Does this PR introduce a user-facing change?

On compatible systems, a mounter's Unmount implementation is changed to not return an error when the specified target can be detected as not a mount point. On Linux, the behavior of detecting a mount point depends on `umount` command is validated when the mounter is created. Additionally, mount point checks will be skipped in CleanupMountPoint/CleanupMountWithForce if the mounter's Unmount having the changed behavior of not returning error when target is not a mount point.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Apr 26, 2022

Please note that we're already in Test Freeze for the release-1.24 branch. This means every merged PR has to be cherry-picked into the release branch to be part of the upcoming v1.24.0 release.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. labels Apr 26, 2022
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Apr 26, 2022

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Apr 26, 2022
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Apr 26, 2022

@cartermckinnon: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Apr 26, 2022
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Apr 26, 2022

Hi @cartermckinnon. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Apr 26, 2022
@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Apr 26, 2022
@@ -244,6 +244,11 @@ func (mounter *Mounter) IsLikelyNotMountPoint(file string) (bool, error) {
return true, nil
}

// CanSafelySkipMountPointCheck always returns false on Windows
func (mounter *Mounter) CanSafelySkipMountPointCheck() bool {
return false
Copy link
Author

@cartermckinnon cartermckinnon Apr 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used false here to avoid changing any behavior on Windows; but looking at the Unmount implementation, I think this might be fine to return true?

Copy link
Contributor

@wongma7 wongma7 Apr 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ya I see the same, it won't make a difference since windows mounter.List does nothing and has a todo

// List returns a list of all mounted filesystems. todo
. i would leave it false, it is up to whoever eventually implements that List todo to figure out when/if CanSafelySkipMountPointCheck should ever return true

@k8s-triage-robot
Copy link

k8s-triage-robot commented Apr 26, 2022

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

@wongma7
Copy link
Contributor

wongma7 commented Apr 26, 2022

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 26, 2022
@@ -303,6 +339,9 @@ func (mounter *Mounter) Unmount(target string) error {
// Rewrite err with the actual exit error of the process.
err = &exec.ExitError{ProcessState: command.ProcessState}
}
if mounter.withSafeNotMountedBehavior && strings.Contains(string(output), errNotMounted) {
Copy link
Contributor

@wongma7 wongma7 Apr 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this in the right place? if umount exits 1 I think err will be non-nil so this belongs in the above if statement. Will need to test.

  1. create Pod with a volume
  2. ssh onto Node and unmount one of its volumes
  3. delete Pod
  4. does this condition actually get hit?

Copy link
Author

@cartermckinnon cartermckinnon Apr 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is within the if err != nil {} block; and I think it should come after the ErrNoChildProcess check (as is). I'll do that test, to confirm.

Copy link
Contributor

@jingxu97 jingxu97 Jun 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put a log here, and update Unmount comment.

Since we are changing the library here, we also need to make sure this behavior change will not affect any other logic.

Copy link
Contributor

@jingxu97 jingxu97 Jun 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

many in-tree drivers uses this function, could you double check this behavior change will not affect any of them?

Copy link
Contributor

@jingxu97 jingxu97 Jun 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also if mount is bad/ corrupted, is it possible that "unmount" returns message "is not a mount point"?

Copy link
Author

@cartermckinnon cartermckinnon Jun 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have all the context on what qualifies as corrupted; but all such not mounted errors will be silenced after this PR (if the mounter supports it). I can't find any in-tree drivers that rely on this error message (or any specific error message), so I'm not concerned about silencing it.

Copy link
Author

@cartermckinnon cartermckinnon Jun 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a quick search, I think corrupted mounts often result in failed: Operation not permitted from umount. That behavior won't change with this PR, the error will be returned.

@cartermckinnon
Copy link
Author

cartermckinnon commented Apr 26, 2022

/retest

@cartermckinnon
Copy link
Author

cartermckinnon commented Apr 27, 2022

I'm not able to reproduce this test failure using:

go test -run ^Test_Run_OneVolumeDetachFailNodeWithReadWriteOnce$ k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler

The line immediately preceding the failed assertion is a sleep statement: https://github.com/cartermckinnon/kubernetes/blob/master/pkg/controller/volume/attachdetach/reconciler/reconciler_test.go#L655

I'm going to try it again, but this seems like flakiness that isn't related to the changes in this PR.

@cartermckinnon
Copy link
Author

cartermckinnon commented Apr 27, 2022

/retest

@k8s-ci-robot k8s-ci-robot removed the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Apr 27, 2022
@cartermckinnon
Copy link
Author

cartermckinnon commented Jun 15, 2022

/retest

@cartermckinnon
Copy link
Author

cartermckinnon commented Jun 22, 2022

@jingxu97 Can you take another look at this?

@cartermckinnon
Copy link
Author

cartermckinnon commented Jun 30, 2022

The only notable change in behavior I see is:

Before this PR, we may try to delete a target that is still a mount point, raising an error. This would happen if the wrong extensiveMountPointCheck value is used (false instead of true). After this PR, we will Unmount the target before attempting to delete it, resulting in expected behavior -- the mount will be successfully cleaned up and no error will be returned.

I can't find any usage of CleanupMountPoint/CleanupMountWithForce that is relying on a specific error being returned; I don't see any concerns around silencing the not mounted error.

For future reference, the only in-tree packages using the extensive mount point check (consistent read) for CleanupMountPoint are:

  • pkg/volume/nfs
  • pkg/volume/local
  • pkg/volume/util/subpath

And the only package using CleanupMountWithForce is pkg/volume/nfs.

@jingxu97
Copy link
Contributor

jingxu97 commented Jun 30, 2022

The only notable change in behavior I see is:

Before this PR, we may try to delete a target that is still a mount point, raising an error. This would happen if the wrong extensiveMountPointCheck value is used (false instead of true). After this PR, we will Unmount the target before attempting to delete it, resulting in expected behavior -- the mount will be successfully cleaned up and no error will be returned.

I can't find any usage of CleanupMountPoint/CleanupMountWithForce that is relying on a specific error being returned; I don't see any concerns around silencing the not mounted error.

For future reference, the only in-tree packages using the extensive mount point check (consistent read) for CleanupMountPoint are:

  • pkg/volume/nfs
  • pkg/volume/local
  • pkg/volume/util/subpath

And the only package using CleanupMountWithForce is pkg/volume/nfs.

Could you also include this behavior change in release note?

@cartermckinnon
Copy link
Author

cartermckinnon commented Jun 30, 2022

@jingxu97 I've updated the release note, let me know if it makes sense 😄

@jingxu97
Copy link
Contributor

jingxu97 commented Jul 1, 2022

inappropriately

The main thing I want to put in the release note is related to the umount function in mount util

@jingxu97 I've updated the release note, let me know if it makes sense 😄

The main thing I want to mention in the release note is about Unmount function in mount-util. Before if it is not a mountpoint, it will return error. After this change, if skipmountpointcheck is true, Unmount will succeed.

Explicit (not sure this is accurate word) mount point checks will now be skipped during CleanupMountPoint/CleanupMountWithForce if "not a mount point" error could be returned when executing umount operation. Also note that before this change, an error would be returned when using Unmount function in mount util. After this change if skipmountpointcheck is true, Unmount will succeed if the path is not a mount point.

@cartermckinnon
Copy link
Author

cartermckinnon commented Jul 1, 2022

The main thing I want to mention in the release note is about Unmount function in mount-util.

I edited a little bit, pls feel free to edit it again

On compatible systems, a mounter's Unmount implementation is changed to not return an error when the specified target can be detected as not a mount point. On Linux, the behavior of detecting a mount point depends on `umount` command is validated when the mounter is created. Additionally, mount point checks will be skipped in CleanupMountPoint/CleanupMountWithForce if the mounter's Unmount having the changed behavior of not returning error when target is not a mount point.

@cartermckinnon
Copy link
Author

cartermckinnon commented Jul 8, 2022

@jingxu97 edits look great to me, I think we're good here?

@jfarrell
Copy link

jfarrell commented Jul 12, 2022

@jingxu97 any pending updates you are waiting on for this for final review/sign-off or are we all set to be able to merge this?

@jingxu97
Copy link
Contributor

jingxu97 commented Jul 14, 2022

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 14, 2022
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jul 14, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cartermckinnon, jingxu97, manugupt1, wongma7

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 14, 2022
@jingxu97
Copy link
Contributor

jingxu97 commented Jul 14, 2022

Thank you very much for the great effort on this PR! Sorry for the delayed review.

@k8s-triage-robot
Copy link

k8s-triage-robot commented Jul 14, 2022

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

  • The PR does have any do-not-merge/* labels
  • The PR does not have the needs-ok-to-test label
  • The PR is mergeable (does not have a needs-rebase label)
  • The PR is approved (has cncf-cla: yes, lgtm, approved labels)
  • The PR is failing tests required for merge

You can:

/retest

@k8s-ci-robot k8s-ci-robot merged commit 7ac06e1 into kubernetes:master Jul 14, 2022
15 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.25 milestone Jul 14, 2022
nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Jul 18, 2022
Recently the k8s.io/mount-utils package added more runtime dectection.
When creating a new Mounter, the detect is run every time. This is
unfortunate, as it logs a message like the following:

```
mount_linux.go:283] Detected umount with safe 'not mounted' behavior
```

This message might be useful, so it probably good to keep it.

In Ceph-CSI there are various locations where Mounter instances are
created. Moving that to the DefaultNodeServer type reduces it to a
single place. Some utility functions need to accept the additional
parameter too, so that has been modified as well.

See-also: kubernetes/kubernetes#109676
Signed-off-by: Niels de Vos <ndevos@redhat.com>
@cartermckinnon cartermckinnon deleted the optimize-unmounts branch Jul 18, 2022
nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Jul 19, 2022
Recently the k8s.io/mount-utils package added more runtime dectection.
When creating a new Mounter, the detect is run every time. This is
unfortunate, as it logs a message like the following:

```
mount_linux.go:283] Detected umount with safe 'not mounted' behavior
```

This message might be useful, so it probably good to keep it.

In Ceph-CSI there are various locations where Mounter instances are
created. Moving that to the DefaultNodeServer type reduces it to a
single place. Some utility functions need to accept the additional
parameter too, so that has been modified as well.

See-also: kubernetes/kubernetes#109676
Signed-off-by: Niels de Vos <ndevos@redhat.com>
nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Jul 20, 2022
Recently the k8s.io/mount-utils package added more runtime dectection.
When creating a new Mounter, the detect is run every time. This is
unfortunate, as it logs a message like the following:

```
mount_linux.go:283] Detected umount with safe 'not mounted' behavior
```

This message might be useful, so it probably good to keep it.

In Ceph-CSI there are various locations where Mounter instances are
created. Moving that to the DefaultNodeServer type reduces it to a
single place. Some utility functions need to accept the additional
parameter too, so that has been modified as well.

See-also: kubernetes/kubernetes#109676
Signed-off-by: Niels de Vos <ndevos@redhat.com>
nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Jul 20, 2022
Recently the k8s.io/mount-utils package added more runtime dectection.
When creating a new Mounter, the detect is run every time. This is
unfortunate, as it logs a message like the following:

```
mount_linux.go:283] Detected umount with safe 'not mounted' behavior
```

This message might be useful, so it probably good to keep it.

In Ceph-CSI there are various locations where Mounter instances are
created. Moving that to the DefaultNodeServer type reduces it to a
single place. Some utility functions need to accept the additional
parameter too, so that has been modified as well.

See-also: kubernetes/kubernetes#109676
Signed-off-by: Niels de Vos <ndevos@redhat.com>
mergify bot pushed a commit to ceph/ceph-csi that referenced this pull request Jul 21, 2022
Recently the k8s.io/mount-utils package added more runtime dectection.
When creating a new Mounter, the detect is run every time. This is
unfortunate, as it logs a message like the following:

```
mount_linux.go:283] Detected umount with safe 'not mounted' behavior
```

This message might be useful, so it probably good to keep it.

In Ceph-CSI there are various locations where Mounter instances are
created. Moving that to the DefaultNodeServer type reduces it to a
single place. Some utility functions need to accept the additional
parameter too, so that has been modified as well.

See-also: kubernetes/kubernetes#109676
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants