Wait for all informers to sync in /readyz.

[wojtek-t]

Based on #92508 by @mborsz

What type of PR is this?
/kind bug

What this PR does / why we need it:
Context: #92506

In large clusters, informers in kube-apiserver may require significant time (30-50s) to initialize. Before this happens, kube-apiserver is not able to answer some requests (e.g. node authorizer is not able to accept any request).

This PR adds a way to determine if informers in kube-apiserver are already synced. This check is added to /readyz which can be used to determine if traffic can be sent to particular kube-apiserver.

Ref: #92506

Extend kube-apiserver /readyz with new "informer-sync" check ensuring that internal informers are synced.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apiserver/OWNERS [wojtek-t]
• test/OWNERS [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wojtek-t]

To copy some context from the original PR:

@deads2k

As I recall, we actually had this check in openshift and chose to remove it because it wasn't required for request safety and answering some requests generally worked better than answering no requests and failing a health check. /readyz would be less worrying from a "death cycle" point of view, but @sttts may still have opinions since he's currently looking at those.

@sttts

Readyz sounds much better than healthz. This is not about being healthy. Would like to hear @logicalhan 's opinion. He reworked the logic there quite a bit few months ago.

@logicalhan

Yeah /readyz sounds appropriate for this. We should probably add a release note stating that we are adding a readyz check which further bifurcates behavior from the deprecated healthz endpoint (as a word of warning).

[wojtek-t]

/cc @mborsz - for posterity

[k8s-ci-robot]

@wojtek-t: GitHub didn't allow me to request PR reviews from the following users: -, for, posterity.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @mborsz - for posterity

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wojtek-t]

/retest

[logicalhan]

/lgtm

just a couple minor comments.

[logicalhan]

This should probably be moved up between lines 616-617, since we only want to add the readyz check once.

[wojtek-t]

done

[logicalhan]

If we're going to do this, then we should probably just fully convert this function so that it doesn't return the bool.

if err != nil || status != http.StatusOK {
   t.Fatalf("got %v but wanted 200, error: %v", status, err)
}

[logicalhan]

And then the calls on 121 and 124 don't have to be in conditional form.

[wojtek-t]

changed the function to return error actually

[logicalhan]

this is a huge nit (sorry):

	stopCh := make(chan struct{})
	// Close stopCh to force checking if informers are synced now.
	close(stopCh)

	var informersByStarted map[bool][]string
	for informType, started := range i.sharedInformerFactory.WaitForCacheSync(stopCh) {
         informersByStarted[started] = append(informersByStarted[started], informType.String())
	}
	if len(informersByStarted[false]) != 0 {
		return fmt.Errorf("%d informers have not started yet: %v (but %v informers have started)", len(notStarted), informersByStarted[false], informersByStarted[true])
	}

[wojtek-t]

done

[logicalhan]

/lgtm

[wojtek-t]

@logicalhan sorry - fixed one more typo - if you could re-lgtm

/hold cancel

[logicalhan]

/lgtm

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[SataQiu]

/retest

[k8s-ci-robot]

New changes are detected. LGTM label has been removed.

[wojtek-t]

Just fixed the minor compile error in integration test - reaaplying the label.

[wojtek-t]

/retest

[k8s-ci-robot]

@wojtek-t: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-kind-ipv6	3f68000	link	/test pull-kubernetes-e2e-kind-ipv6

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.