Pods get stuck in ContainerCreating state when iSCSI attaching fails

[tksm]

Describe the bug
Pods get stuck in ContainerCreating state when iSCSI attaching fails. It is never automatically recovered once it happens.

If I understand correctly, iSCSI attaching is done by the NodeStageVolume call, and the kubelet will retry if it fails. But the trident log shows the NodeStageVolume call returned success and never be retried, although the iSCSI attaching failed. The subsequent NodePublishVolume call failed to mount and returned an internal error.

I found that the trident ignores errors other than auth errors from utils.AttachISCSIVolume(). This change seems to be introduced in 22.04.0. I think this change may cause this issue.

trident/frontend/csi/node_server.go

Lines 1046 to 1060 in e721974

	// Perform the login/rescan/discovery/(optionally)format, mount & get the device back in the publish info
	if err := utils.AttachISCSIVolume(ctx, req.VolumeContext["internalName"], "", publishInfo); err != nil {
	// Did we fail to log in?
	if utils.IsAuthError(err) {
	// Update CHAP info from the controller and try one more time
	Logc(ctx).Warn("iSCSI login failed; will retrieve CHAP credentials from Trident controller and try again.")
	if err = p.updateChapInfoFromController(ctx, req, publishInfo); err != nil {
	return nil, status.Error(codes.Internal, err.Error())
	}
	if err = utils.AttachISCSIVolume(ctx, req.VolumeContext["internalName"], "", publishInfo); err != nil {
	// Bail out no matter what as we've now tried with updated credentials
	return nil, status.Error(codes.Internal, err.Error())
	}
	}
	}

Environment
Provide accurate information about the environment to help us reproduce the issue.

• Trident version: 22.04.0
• Trident installation flags used: silenceAutosupport: true (Trident Operator)
• Container runtime: containerd://1.4.13
• Kubernetes version: v1.23.4
• Kubernetes orchestrator: Kubernetes
• Kubernetes enabled feature gates: none
• OS: Ubuntu 20.04.4 LTS
• NetApp backend types: ONTAP AFF 9.9.1P
• Other:

To Reproduce
Steps to reproduce the behavior:

1. Make one node unable to connect LIFs.
   • iptables -A OUTPUT -p tcp -d <LIF_ADDRESS> -j REJECT
2. Create a StatefulSet that has an ontap-san volume on the node.
3. The Pod gets stuck in ContainerCreating.
4. Make the node able to connect LIFs.
   • iptables -D OUTPUT -p tcp -d <LIF_ADDRESS> -j REJECT
5. The Pod still sticks in ContainerCreating.

Expected behavior

The trident should retry the iSCSI attaching when it fails.

Additional context

[gnarl]

This issue is fixed with commit ee934c9 and is included in the Trident 22.07 release.

[tksm]

@gnarl Thank you for fixing this issue. I confirmed the issue is no longer reproduced on Trident v22.07.0. 👍

Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Normal   SuccessfulAttachVolume  72s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-5cb4eb99-6d25-43ed-a495-91e3d51128c4"
  Warning  FailedMount             15s (x3 over 47s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-5cb4eb99-6d25-43ed-a495-91e3d51128c4" : rpc error: code = Internal desc = failed to stage volume: iSCSI login failed
# I confirmed the pod would mount the volume by retrying after making the node able to connect LIFs.
  Normal   Pulling                 11s                kubelet                  Pulling image "nginx"
  Normal   Pulled                  10s                kubelet                  Successfully pulled image "nginx" in 1.517655829s
  Normal   Created                 10s                kubelet                  Created container nginx
  Normal   Started                 10s                kubelet                  Started container nginx