unmount timeout should NOT trigger detach (by trigger detach there is a potential data loose)

[shay-berman]

/kind bug

What happened:
Stop a POD with PV, will trigger unmount flow in k8s. And the current flow is to trigger detach of the PV if the unmount operation reached to timeout (called maxWaitForUnmountDuration, which BTW is not configurable #68086).

I think that this is a bug - k8s should not send detach to PV if the unmount operation is stuck and reached to timeout maxWaitForUnmountDuration. Because there is a small chance of data loose here, because there is a chance that the unmount operation didn't unmount the device on the node, and some data still preserve in the filesystem cache.

What you expected to happen:
K8s should not send detach to the PV if the unmount of PV reached to timeout maxWaitForUnmountDuration. Instead it should just fail the unmount and just retry again the unmount.

How to reproduce it (as minimally and precisely as possible):

1. change the unmount operation to a fake one that just goes to endless loop.
2. create POD with PV on the host with the fake unmount.
3. delete the POD -> after few minutes (timeout) k8s should send detach to the PV on this host because the unmount was stuck.

Anything else we need to know?:
We had a discussion about how k8s discover a crashed node and release the PODs on it (HERE ->
#65392 (comment))
Is some way this current issue is related(complimentary) to this one.

Environment:
every k8s version.

[mlmhl]

/sig storage

[kasisnu]

Hi. I think I see the problem with this. Can I volunteer to work on a fix?

New to the k8s codebase

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kmova]

/remove-lifecycle rotten

[kmova]

/reopen

[k8s-ci-robot]

@kmova: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.