Refresh discovery server resources for memCacheClient in parallel

[jpbetz]

What type of PR is this?

/kind flake

What this PR does / why we need it:

Run discovery API calls in parallel to reduce for in-memory discovery client. Reduces each cache refresh from ~15 seconds to ~ 2 seconds.

This helps e2e tests since they are run in parallel and some remove CRDs at the same others are being run, forcing the cache to be refreshed often.

In e2e tests we're seeing a groupVersion list length of around 27, which means we are 27 API calls in series.

See #87668 and #88297 for debugging details.

There might be other ways to optimize this further, but running the requests in parallel is an obvious win, so would like to merge this before pursuing other optimizations.

Last 9 pull-kubernetes-e2e-gce runs below. No GC failures. This is with SSA at 10%

1230671712341725185 - pass - 53m18s
1230729716571312129 - pass - 49m35s
1230742932269568000 - pass - 47m0s
1230756271573962752 - pass - 45m38s
1230767966954459136 - pass - 51m17s
1230781434029936640 - 12 In-tree Volumes failures - 48m41s
1230885123486912512 - 1 volume mount failure - 50m28s
1230899588768993280 - 1 volume mount failure - 51m12s
1230916073155465216 - pass - 49m40s

From #88297 here are the last 9 runs with SSA at 100%. No GC failures.

1230746203231096836 - pass - 1h1m41s
1230762058010595328 - 1 volume/CSIDriver failure - 1h0m17s
1230778298858999808 - 15 In-tree Volumes failures - 50m16s
1230885241724342272 - pass - 51m50s
1230899588773187585 - pass - 52m36s
1230919719050022912 - pass - 57m56s
1230949943569551360 - 1 In-tree Volumes failures, 2 PersistentVolumes-local failures - 59m57s
1230966546394779648 - 1 CRD publish OpenAPI failure, 58m36s
1230985169494609921 - pass - 55m29s

Which issue(s) this PR fixes:

Fixes #87668

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[apelisse]

Thanks
/lgtm

[jpbetz]

e2e test are failing, requests are getting throttled:

W0220 21:49:48.390781       1 garbagecollector.go:661] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
 I0220 21:49:48.390954       1 garbagecollector.go:203] syncing garbage collector with updated resources from discovery (attempt 5): added: [/v1, Resource=configmaps /v1, Resource=endpoints /v1, Resource=events /v1, Resource=limitranges /v1, Resource=nam\
 espaces /v1, Resource=nodes /v1, Resource=persistentvolumeclaims /v1, Resource=persistentvolumes /v1, Resource=pods /v1, Resource=podtemplates /v1, Resource=replicationcontrollers /v1, Resource=resourcequotas /v1, Resource=secrets /v1, Resource=servicea\
 ccounts /v1, Resource=services admissionregistration.k8s.io/v1, Resource=mutatingwebhookconfigurations admissionregistration.k8s.io/v1, Resource=validatingwebhookconfigurations apiextensions.k8s.io/v1, Resource=customresourcedefinitions apiregistration.\
 k8s.io/v1, Resource=apiservices apps/v1, Resource=controllerrevisions apps/v1, Resource=daemonsets apps/v1, Resource=deployments apps/v1, Resource=replicasets apps/v1, Resource=statefulsets autoscaling/v1, Resource=horizontalpodautoscalers batch/v1, Res\
 ource=jobs batch/v1beta1, Resource=cronjobs certificates.k8s.io/v1beta1, Resource=certificatesigningrequests coordination.k8s.io/v1, Resource=leases discovery.k8s.io/v1beta1, Resource=endpointslices events.k8s.io/v1beta1, Resource=events extensions/v1be\
 ta1, Resource=ingresses networking.k8s.io/v1, Resource=networkpolicies networking.k8s.io/v1beta1, Resource=ingresses node.k8s.io/v1beta1, Resource=runtimeclasses policy/v1beta1, Resource=poddisruptionbudgets policy/v1beta1, Resource=podsecuritypolicies \
 rbac.authorization.k8s.io/v1, Resource=clusterrolebindings rbac.authorization.k8s.io/v1, Resource=clusterroles rbac.authorization.k8s.io/v1, Resource=rolebindings rbac.authorization.k8s.io/v1, Resource=roles scalingpolicy.kope.io/v1alpha1, Resource=scal\
 ingpolicies scheduling.k8s.io/v1, Resource=priorityclasses settings.k8s.io/v1alpha1, Resource=podpresets snapshot.storage.k8s.io/v1beta1, Resource=volumesnapshotclasses snapshot.storage.k8s.io/v1beta1, Resource=volumesnapshotcontents snapshot.storage.k8\
 s.io/v1beta1, Resource=volumesnapshots storage.k8s.io/v1, Resource=csinodes storage.k8s.io/v1, Resource=storageclasses storage.k8s.io/v1, Resource=volumeattachments storage.k8s.io/v1beta1, Resource=csidrivers], removed: []

[jpbetz]

Need to prevent client side rate limiter induced timeouts. Trying to limit concurrency to 5 to see if that helps.

[jpbetz]

I misunderstood the error, it looks like there is a correctness problem. Fixing.

[jpbetz]

/test pull-kubernetes-e2e-gce

two passes in a row, going to try a few more

[jpbetz]

/test pull-kubernetes-e2e-gce

[jpbetz]

/test pull-kubernetes-e2e-gce

[jpbetz]

/test pull-kubernetes-e2e-gce

[jpbetz]

/test pull-kubernetes-e2e-gce

[jpbetz]

@liggitt, @apelisse I think this is ready to go. Would you have a look?

[jpbetz]

cc @lavalamp @caesarxuchao

[apelisse]

/lgtm
Thanks!

[liggitt]

/approve

two nits, then lgtm

[liggitt]

/hold for nits

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/client-go/OWNERS [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jpbetz]

/hold cancel
nits fixed

[apelisse]

/lgtm
Thanks!

[lavalamp]

var resultLock sync.Mutex

works fine here, I think? No need to change it, we'll just wait for an "ineffassign" fix...

[lavalamp]

/lgtm

(I want credit for reading it ;) )

[lavalamp]

/lgtm