PVC attachment timeout when a node is terminated / removed from the cluster

[paalkr]

Describe the bug
It looks like this bug #691 or a similar bug is introduced in Trident after the 22.01.1 release. Both the 22.04.0 and the 22.07.0 releases suffer from the same behavior.

Environment
Vanilla Kubernetes 1.21.12 deployed to AWS using kOps 1.23.1

Trident version: [e.g. 19.10]
22.04.0 and 22.07.1
Trident installation flags used: [e.g. -d -n trident --use-custom-yaml]
helm install -n kube-netapp trident .\trident-operator-chart-21.07\ -f .\values.yaml
Container runtime: [e.g. Docker 19.03.1-CE]
Docker 20.10.11
Kubernetes version: [e.g. 1.15.1]
1.21.12
Kubernetes orchestrator: [e.g. OpenShift v3.11, Rancher v2.3.3]
kOps 1.23.1
Kubernetes enabled feature gates: [e.g. CSINodeInfo]
None
OS: [e.g. RHEL 7.6, Ubuntu 16.04]
Ubuntu 20.04.4 LTS
NetApp backend types: [e.g. CVS for AWS, ONTAP AFF 9.5, HCI 1.7]
AWS FSx for NetApp ONTAP, ontap-san-economy driver

Other:
Expected behavior
PVCs to attach after the initial 6 minutes delay.

Trident Backend config

apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: fsx01-k8s11-san-eco
spec:
  version: 1
  storageDriverName: ontap-san-economy
  managementLIF: svm-xxxx.fs-xxxx.fsx.eu-north-1.amazonaws.com
  debug: true
  svm: k8s11
  storagePrefix: k8s11
  useCHAP: true
  credentials: 
    name: fsx-netapp
  defaults:
    tieringPolicy: auto

[gnarl]

Hi @paalkr,

Can you provide details on how long the detach is actually taking? Kubernetes is eventually consistent so the 6 minute expectation may not be reasonable. Also, providing Trident log snippets and other observations can help us troubleshoot this issue.

[paalkr]

Hi, thanks for the feedback. I think the main problem is that the PVC isn't detached properly when a node is just terminated. When a replacement node is started and added to the cluster, and the workload rescheduled the PVC is stuck in the multi attach error state indefinitely. Trident thinks the PVC is still attached to the terminated node.

Which logs would be the most helpful? The attacher container logs? Trident operator? Trident-main container on the daemonset?

[gnarl]

Hi @paalkr,

This isn't something that Trident can currently control. It isn't just Trident that believes that the volume might still be attached. The Kubernetes Attach/Detach Controller also believes that the volume might still be attached. If there were a network failure then the K8S worker node may be unreachable but the volume could still be attached. The concept of adding "Forced Detach" support to Kubernetes hasn't been implemented yet. "Forced Detach" allows the I/O to be shutoff by the backing storage prior to rescheduling the Pod to avoid data corruption.

We've tested your originally reported scenario recently and observed that it took about 6 minutes to reschedule the Pod. This agrees with what you have in the opening comment of this issue.

[paalkr]

The problem is that for every release after 22.01.1 the PVCs are no longer attached after the 6 minutes timeout. The PVCs stays unattached indefinitely.

With the 22.01.1 release the PVCs are attached after 6 minutes, as described in the initial issue her: #691

[paalkr]

Any news on this? The issue is a huge problem for us and the main blocker for not folly rolling out AWS FSx for NetApp as our main storage provider in a +5000 pod cluster.

[paalkr]

After executing some more tests with the 22.07 driver I now believe that the multi attach issue is in fact handled properly. The issue we se is related to iSCSI PVCs not mounting correctly after being attached to a node.

MountVolume.MountDevice failed for volume "pvc-50aad985-1f5f-45da-8762-26ae500890dd" : rpc error: code = Internal desc = failed to stage volume: no devices present yet

If I restart the CSI driver on the particular node, the PVC mounts correctly.

I will close this issue and create new issue for the mount problem.

[paalkr]

The mount issue was due to errors with the node prep. After fixing the node prep problems the Trident CSI driver seems to behave as it should. Re-attaching and mounting works as expected, even with spot terminated nodes. But apparently multipath are interfering with the EBS CSI driver. I opened a ticket in the EBS CSI driver project. kubernetes-sigs/aws-ebs-csi-driver#1417