orphaned pod flexvolume can not be cleaned up

[xiaoanyunfei]

What happened:
On kubelet node, some pod have been deleted, but the volume of the pod can not be umounted and deleted.

There are always the following log

kubelet_pods.go:1684] Orphaned pod "bb07fd8d-b737-11e9-a203-0022ac9722f5" found, but volumes not yet removed. Reducing cpu to minimum

We use flexvolume, the mount path is
/var/lib/kubelet/pods/bb07fd8d-b737-11e9-a203-0022ac9722f5/volumes/hulk~lvm

I notices some log when starting kubelet

W0805 12:17:35.293525 260621 reconciler.go:394] Could not construct volume information, cleanup the mounts. (pod.UID bb07fd8d-b737-11e9-a203-0022ac9722f5, volume.SpecName hulklvm): no volume plugin matched
I0805 12:17:35.293534 260621 reconciler.go:429] Reconciler sync states: could not find information (PID: bb07fd8d-b737-11e9-a203-0022ac9722f5) (Volume SpecName: hulklvm) in desired state, clean up the mount points
E0805 12:17:35.293571 260621 reconciler.go:442] volumeHandler.UnmountVolumeHandler for UnmountVolume failed for volume "" (UniqueName: "hulklvm") pod "bb07fd8d-b737-11e9-a203-0022ac9722f5" (UID: "bb07fd8d-b737-11e9-a203-0022ac9722f5") : UnmountVolume.FindPluginByName failed for volume "" (UniqueName: "hulklvm") pod "bb07fd8d-b737-11e9-a203-0022ac9722f5" (UID: "bb07fd8d-b737-11e9-a203-0022ac9722f5") : no volume plugin matched

What you expected to happen:

The directory
/var/lib/kubelet/pods/bb07fd8d-b737-11e9-a203-0022ac9722f5/
is cleaned up

How to reproduce it (as minimally and precisely as possible):
one way to reproduce it
1· create a new pod with flexvolume
2· stop kubelet
3· rm container of the pod
4· kubectl delete pod
5· start kubelet

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
  1.11.3
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

/sig storage

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[xiaoanyunfei]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.