Init cache with assigned non-terminated pods before scheduling

[k82cn]

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #45220

Release note:

The fix makes scheduling go routine waiting for cache (e.g. Pod) to be synced.

[k8s-reviewable]

This change is 

[k82cn]

/cc @bsalamat

[k82cn]

I'm going to add some test for it.

[wanghaoran1988]

why not wait scheduledPodPopulator.hasSynced ?

[k82cn]

why not wait scheduledPodPopulator.hasSynced ?

reasonable, let me check the detail of Informer.

[BlueBlue-Lee]

5908108#diff-77bdee21550bc698bfa654de11c64324L455

I think it will never return because the corresponding controller is not started.

[k82cn]

5908108#diff-77bdee21550bc698bfa654de11c64324L455
I think it will never return because the corresponding controller is not started.

Thanks for the comments, addressed.

[k82cn]

Tested in local cluster by following steps:

1. Start cluster with ./hack/local-cluster-up.sh (8 cpu, 2G mem)
2. Submit 10 pods (1 cpu, 300m mem): 7 pod will be running and 3 pods will be pending (dns consumed 260m cpu)
3. Keep restarting "hyperkube scheduler":
   • without fix, it may report OutOfCpu
   • with fix, not status changed (7 running, 3 pending)

The reason is that we did not wait for cache to be synced by before scheduling, so the running pods may not be added into cache when scheduling. We did not need to wait for unassigned tasks, scheduler will get it one by one to dispatch.

/cc @bsalamat @wojtek-t

[bsalamat]

/lgtm

Thanks for the fix, Klaus!

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bsalamat, k82cn
We suggest the following additional approver: wojtek-t

Assign the PR to them by writing /assign @wojtek-t in a comment when ready.

Needs approval from an approver in each of these OWNERS Files:

• cmd/libs/go2idl/OWNERS
• pkg/client/OWNERS
• plugin/cmd/kube-scheduler/OWNERS
• staging/OWNERS

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[timothysc]

@k82cn please add a release note on this one.

@ncdc any comments? It looks good to me, re: prevents a startup race, but does this fix need to propagate to the kcm?

[k82cn]

please add a release note on this one.

Done

re: prevents a startup race, but does this fix need to propagate to the kcm?

I can do some check today and append comments here.

[k82cn]

@ncdc any comments? It looks good to me, re: prevents a startup race, but does this fix need to propagate to the kcm?

@ncdc / @timothysc , just check the kcm: each controller call cache.WaitForCacheSync in Run() (cronjob sync it by List).

[wanghaoran1988]

@k82cn Scheduler seems use the sharedInformerFactory for other resources like nodes,configMap,pv, except for the pod, here is the code for pods: scheduledPodIndexer, c.scheduledPodPopulator = cache.NewIndexerInformer, it's a not registered informer of the sharedInformerFactory, so I think your code sharedInformerFactory,WaitForCacheSync will not sync the scheduled pod and notScheduledPod.
Also, your code fix part of the problem, you synced the other resources except pods, I am not clear why the scheduledPodIndexer and scheduledPodPopulator not using the sharedInformer, maybe we have other concerns ? Correct me if I am wrong.
/cc @timothysc @davidopp

[k82cn]

scheduledPodIndexer, c.scheduledPodPopulator = cache.NewIndexerInformer, it's a not registered informer of the sharedInformerFactory,

Good point! I'll add logic to handle it. We did not add it into informer factory because the ListerWatcher is customized (un-assigned, non-terminated).

[k8s-github-robot]

Automatic merge from submit-queue (batch tested with PRs 45453, 45307, 44987)