Add node check to vSphere cloud provider

[jsafrane]

What type of PR is this?

/kind bug

What this PR does / why we need it:

At KCM startup, vSphere cloud provider builds a cache from NodeAdded() events from an informer. But these events are asynchronous, the cloud provider may need information from the cache earlier, for example when detaching a volume from a node at KCM startup. If a node is missing in the cache, the cloud provider treats such a detach as successful, which is wrong. Such a volume will be attached to the node forever.

To prevent this issue:

1. Try nodeLister from the informer before declaring a node as not found. A/D controller starts after its node informer has been synced.

2. Read API server before declaring a node as not found. To be 100% sure a node has been really deleted and the informer does not have stale data.

Which issue(s) this PR fixes:

Fixes #117091

Special notes for your reviewer:

Since vSphere CSI migration is GA, the vSphere cloud provider is used only to detach volumes from 1.25 and older nodes (unless the migration is disabled).

I've tested this with hacked errors in the cloud provider to fail Detach requests + restarted KCM - with this PR, I can see in logs:

Node jsafrane-8pmvh-worker-xlb2s found in vSphere cloud provider node informer

Which means the node is not in NodeManager's cache and the old cloud provider would treat all volumes used by this node as detached. With this PR, the cloud provider actually detaches volumes from the node.

Does this PR introduce a user-facing change?

Fixed vSphere cloud provider not to skip detach volumes from nodes at kube-controller-startup.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jsafrane]

/assign @gnufied @divyenpatel

[jsafrane]

/priority important-soon

[divyenpatel]

@lubronzhan Can you help review this PR?

[gnufied]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: f44ce987190d7c201b6638febfac85e189f8a8fa

[jsafrane]

/assign @cheftako @andrewsykim
For approval. This is a bug in the in-tree vSphere provider that can get upgrade to out-of-tree CSI driver stuck when KCM is restarted at a wrong time. I plan to backport it to older releases too.

[jsafrane]

/assign @nckturner
for approval

[nckturner]

/approve bug fix

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsafrane, nckturner

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/legacy-cloud-providers/OWNERS [nckturner]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment