pkg/kubelet: improve the node informer sync check

[neolit123]

What this PR does / why we need it:

GetNode() is called in a lot of places including a hot loop in
fastStatusUpdateOnce. Having a poll in it is delaying
the kubelet /readyz status=200 report.

If a client is available attempt to wait for the sync to happen,
before starting the list watch for pods at the apiserver.
This is done to avoid caching of Node objects.

Some test data for a kubeadm setup that manages the kube-apiserver as a static pod -
waiting for the kube-apiserver and kubelet to report 200 at /healthz.

• Without this patch: 72 seconds
• With this patch: 12 seconds
  (it restores the old fast timing / behavior)

Which issue(s) this PR fixes:

xref kubernetes/kubeadm#2395
xref kubernetes/kubelet#23

Special notes for your reviewer:

Does this PR introduce a user-facing change?

kubelet: fixes a performance regression when waiting for a synchronization of the node list with the kube-apiserver

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[neolit123]

/kind regression
(^ debatable for self-hosting kube-apiservers)

/cc @derekwaynecarr @liggitt

[dims]

cc @adisky

[neolit123]

given the problem shown in #99336 (comment)

i'm marking this as release blocking for 1.21.
if someone has objections to this, i'm open to discussion.
/priority critical-urgent
/milestone v1.21

in the meantime we are discussing and trying to fix it.

[ehashman]

/triage accepted

[ehashman]

flake #98856
/retest

[derekwaynecarr]

just to correct my stmt above, static pods do go through kubelet pod admission (mirror pods do not), but by definition static pods need to work well with the kubelet local view of the node, so I think we do not have any risks.

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, neolit123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/kubelet/OWNERS [derekwaynecarr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liggitt]

/lgtm

thanks for all the iterations and care

[neolit123]

thank you to all who helped with review and the discussion.
i can send backports shortly.

apparently there will be another 1.18 PATCH release, so this can be backported there too.

[neolit123]

cherry picks:
#101343
#101344
#101345
#101346

^ need review / LGTM / approval.
diffs involved some adaptation around function argument changes and no structured logging in older versions.
hope i didn't make any mistakes.

all patches tested with a local cluster.
random observation: 1.21 kubelet + apiserver report 200 at healthz faster than e.g. 1.18.

[liggitt]

picks reviewed, need approval from kubelet owners and then release branch acks