GCE PD Detach fails if node no longer exists

[saad-ali]

Problem:

If a node with a GCE PD attached is deleted (before the volume is detached), subsequent attempts by the attach/detach controller to detach it continuously fail, and prevent the controller from attaching the volume to another node.

Repro steps:

1. Create a pod referencing a GCE PD
2. Wait for pod to get scheduled and running.
3. Delete the node VM (using gcloud) that the pod is scheduled to.
4. Check if volume is detached by attach/detach controller:
   • Expected: Volume detached.
   • Actual: Volume continuously fails detach.

Logs:

From `/var/log/kube-controller-manager.log`:

I0721 03:21:43.464087       7 reconciler.go:134] Started DetachVolume for volume "X" from node "Y" due to maxWaitForUnmountDuration expiry.
E0721 03:21:43.591941       7 gce.go:2580] getInstanceByName/single-zone: failed to get instance Y; err: googleapi: Error 404: The resource 'projects/[project]/zones/[zone]/instances/Y' was not found, notFound
E0721 03:21:43.591985       7 attacher.go:215] Error checking if PD ("[pdname]") is already attached to current node ("Y"). Will continue and try detach anyway. err=instance not found
E0721 03:21:43.698786       7 gce.go:2580] getInstanceByName/single-zone: failed to get instance Y; err: googleapi: Error 404: The resource 'projects/[project]/zones/[zone]/instances/Y' was not found, notFound
E0721 03:21:43.698828       7 attacher.go:225] Error detaching PD "[pdname]" from node "Y": error getting instance "Y"

Workarounds:

• Restart the kube-controller-manager binary on the master.

-or-

• Recreate a node with the same name.

Proposed Fix:

If GCE PD detach fails with instance not found, assume successful detach.

[pmorie]

So is the issue here that we made a bad assumption about how the GCE API will behave for detach when the node doesn't exist?

[saad-ali]

Not so much a bad assumption, more a missed case.

[saad-ali]

The fix for this, PR #29485, missed one location where node is fetched. The existing fix handles the case where the actual node is physically deleted. But it does not handle the case where the node API object is deleted. This means that detach can still sometimes fail due to missing node API object:

Error: DetachVolume failed fetching node from API server for volume "kubernetes.io/gce-pd/panda-disk" (spec.Name: "data") from node "nodA" with: nodes "nodeA" not found