FailedMount: Unable to attach or mount volumes

[glennswest]

Describe the bug
In a openshift 4.4.5 bare mental install with trident, we have seen several cases where a pod getting unscheduled from one worker, and scheduled to another, will get in a state where the new pod cannot attach or mount the volume.

The current work around is to restart the kubelets. This is for large scale install, and this instability is getting to be critical.

The logs look like the following:

time="2020-11-03T14:21:03Z" level=debug msg="GRPC call: /csi.v1.Node/NodeUnpublishVolume"
time="2020-11-03T14:21:03Z" level=debug msg="GRPC request: volume_id:"pvc-9455721f-3a5e-49e1-9844-713bbcb3f5a8" target_path:"/var/lib/kubelet/pods/0a61b028-214e-4074-8443-258f74b8b91b/volumes/kubernetes.iocsi/pvc-9455721f-3a5e-49e1-9844-713bbcb3f5a8/mount" "
time="2020-11-03T14:21:03Z" level=debug msg="Attempting to acquire shared lock (NodeUnpublishVolume-pvc-9455721f-3a5e-49e1-9844-713bbcb3f5a8)." lock=csi_node_server
time="2020-11-03T14:21:03Z" level=debug msg="Acquired shared lock (NodeUnpublishVolume-pvc-9455721f-3a5e-49e1-9844-713bbcb3f5a8)." lock=csi_node_server
time="2020-11-03T14:21:03Z" level=debug msg=">>>> NodeUnpublishVolume" Method=NodeUnpublishVolume Type=CSI_Node
time="2020-11-03T14:21:03Z" level=debug msg="<<<< NodeUnpublishVolume" Method=NodeUnpublishVolume Type=CSI_Node
time="2020-11-03T14:21:03Z" level=debug msg="Released shared lock (NodeUnpublishVolume-pvc-9455721f-3a5e-49e1-9844-713bbcb3f5a8)." lock=csi_node_server
time="2020-11-03T14:21:03Z" level=error msg="GRPC error: rpc error: code = Internal desc = could not check if the target path (/var/lib/kubelet/pods/0a61b028-214e-4074-8443-258f74b8b91b/volumes/kubernetes.iocsi/pvc-9455721f-3a5e-49e1-9844-713bbcb3f5a8/mount) is a directory; stat /var/lib/kubelet/pods/0a61b028-214e-4074-8443-258f74b8b91b/volumes/kubernetes.io~csi/pvc-9455721f-3a5e-49e1-9844-713bbcb3f5a8/mount: stale NFS file handle"
A clear and concise description of what the bug is.

Environment
Provide accurate information about the environment to help us reproduce the issue.

• Trident version: [e.g. 19.10]
  20.07.01
• Trident installation flags used: [e.g. -d -n trident --use-custom-yaml]

OpenShift 4.4.5

• Container runtime: [e.g. Docker 19.03.1-CE]
• Kubernetes version: [e.g. 1.15.1]
• Kubernetes orchestrator: [e.g. OpenShift v3.11, Rancher v2.3.3]
• Kubernetes enabled feature gates: [e.g. CSINodeInfo]
• OS: [e.g. RHEL 7.6, Ubuntu 16.04]
  RHCOS
• NetApp backend types: [e.g. CVS for AWS, ONTAP AFF 9.5, HCI 1.7]
• Other:

To Reproduce
Steps to reproduce the behavior:
This appears to happen at random.
Im currently doing a specific test case of a container that switches between nodes to be able to reproduce it easier.

Expected behavior
A clear and concise description of what you expected to happen.
Pods have no trouble moving between nodes.

Additional context
Add any other context about the problem here.

[gnarl]

Hi @glennswest,

This error "GRPC error: rpc error: code = Internal desc = could not check if the target path (/var/lib/kubelet/pods/0a61b028-214e-4074-8443-258f74b8b91b/volumes/kubernetes.iocsi/pvc-9455721f-3a5e-49e1-9844-713bbcb3f5a8/mount) is a directory;" was fixed with commit f59a30. The fix is included in the Trident v20.10.0 release.

Did you recently upgrade Kubernetes in your cluster?

[gnarl]

Hi @glennswest,

We just reread this issue and see that you are running Trident v20.07.1 which also has that fix. The failure to mount is then likely to be one of the two following scenarios.

1. The node where the attachment is being made doesn't have access to the storage backend. Access the node and confirm network connectivity exists between the node and the ONTAP DataLIF. If connectivity exists then attempt to run the mount command from a shell on the node.
2. Also, verify that worker node preparation was completed successfully on the failing node. Do this by checking for installed packages and verifying that required services are running.

If this still does not resolve your issue please contact NetApp Support.

[glennswest]

The nodes are operating normally prior to this. No upgrade was done. Clean new install. All the workers are prepared the same, and the deployment of trident is via operator. Restarting kubelet manually on both the "original" node and the "new" node clears the error. This has happend several times.

[gnarl]

Hi @glennswest,

You'll want to open a NetApp Support case so that we can assist you with further problem determination.

To open a case with NetApp, please go to https://mysupport.netapp.com/site/.

• Bottom left, Click on 'Contact Support'
• Find the appropriate number from your region to call in, or login.
• Note: Trident is a supported product by NetApp based on a supported Netapp storage SN.
• Open the case on the NetApp storage SN, and provide the description of the problem.
• Be sure to mention the product is Trident on Kubernetes, and provide the details. Mention this GitHub.
• The case will be directed to Trident support engineers for response.

[tsmetana]

This error "GRPC error: rpc error: code = Internal desc = could not check if the target path (/var/lib/kubelet/pods/0a61b028-214e-4074-8443-258f74b8b91b/volumes/kubernetes.iocsi/pvc-9455721f-3a5e-49e1-9844-713bbcb3f5a8/mount) is a directory;" was fixed with commit f59a30. The fix is included in the Trident v20.10.0 release.

Hello. I think this is not completely correct. Look carefully at the error message and the patch in f59a30. The error we're encountering corresponds to the else branch of the test that was changed. Note the file we're accessing exists but is a stale filesystem handle so stat would end with a differrent error.

[gnarl]

@glennswest and @tsmetana,

Were you able to fix your issue where pods were not mounting the volume?

[glennswest]

Last report I had, when the customer upgraded, the problem appears to be solved. Know more in 2021.

[gnarl]

Closing this issue as no additional updates have been reported.