kubelet: storage: don't hang kubelet on unresponsive nfs

[sjenning]

Fixes #31272

Currently, due to the nature of nfs, an unresponsive nfs volume in a pod can wedge the kubelet such that additional pods can not be run.

The discussion thus far surrounding this issue was to wrap the lstat, the syscall that ends up hanging in uninterruptible sleep, in a goroutine and limiting the number of goroutines that hang to one per-pod per-volume.

However, in my investigation, I found that the callsites that request a listing of the volumes from a particular volume plugin directory don't care anything about the properties provided by the lstat call. They only care about whether or not a directory exists.

Given that constraint, this PR just avoids the lstat call by using Readdirnames() instead of ReadDir() or ReadDirNoExit()

More detail for reviewers

Consider the pod mounted nfs volume at /var/lib/kubelet/pods/881341b5-9551-11e6-af4c-fa163e815edd/volumes/kubernetes.io~nfs/myvol. The kubelet wedges because when we do a ReadDir() or ReadDirNoExit() it calls syscall.Lstat on myvol which requires communication with the nfs server. If the nfs server is unreachable, this call hangs forever.

However, for our code, we only care what about the names of files/directory contained in kubernetes.io~nfs directory, not any of the more detailed information the Lstat call provides. Getting the names can be done with Readdirnames(), which doesn't need to involve the nfs server.

@pmorie @eparis @ncdc @derekwaynecarr @saad-ali @thockin @vishh @kubernetes/rh-cluster-infra

---

This change is

[derekwaynecarr]

@k8s-bot gci gke e2e test this

[derekwaynecarr]

@sjenning -- per our chat, a comment that describes the problem on this pr with an actual example set of directories was helpful for simpletons like myself to follow the core problem.

[derekwaynecarr]

@k8s-bot gci gke e2e test this

[sjenning]

@derekwaynecarr I updated the first comment with a more detailed example of how this happens IRL

[derekwaynecarr]

@k8s-bot gci gke e2e test this

[derekwaynecarr]

LGTM

[derekwaynecarr]

fyi @kubernetes/sig-node

[wongma7]

any chance of this fix being cherry-picked?

[derekwaynecarr]

@k8s-bot gci gke e2e test this

[k8s-cherrypick-bot]

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

[k8s-github-robot]

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

[k8s-ci-robot]

Jenkins GCI GKE smoke e2e failed for commit da3683e. Full PR test history.

The magic incantation to run this job again is @k8s-bot gci gke e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

[k8s-github-robot]

Automatic merge from submit-queue

[saad-ali]

Thanks for digging into this and getting a workaround out for 1.5 @sjenning!

Let's get this cherry-picked back to 1.4.

[jingxu97]

@sjenning, during some testing, I noticed that your fix still won't solve the problem of kubelet hung on unresponsive nfs. When a node has some directories mounting to a nfs server (running in a container) exports, after the nfs server container is deleted, ReadDirNoStat() function will hung too.

Could you please check again and see whether you get different result? Thank you!

cc @kubernetes/sig-storage