gci-gce-scalability flakes - get configmaps latency issue

[krzysied]

Which jobs are failing:
gci-gce-scalability - this is 100 nodes test for gce. This test is flaking.
Which test(s) are failing:
Density test

Since when has it been failing:
First fail can be seen in run 23438

Testgrid link:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/23438

Reason for failure:
99th percentile of GET configmaps call latency is greater than 1s.

Anything else we need to know:

[krzysied]

/sig scalability

[krzysied]

It looks like the problem is related to the increased number of get configmaps call. When the get configmaps calls happen at the same time as saturation pods creation, their latencies significantly increase.

IIUC the increased number of get configmaps calls is due to #74755

[krzysied]

#74755 should be only a temporary change. I expect that get configmap latency issue should resolve on its own.

/assign
I will try to debug it and find the bottleneck anyway.

[krzysied]

/cc @mm4tt
/cc @wojtek-t

[wojtek-t]

This #74781 should hopefully get it back to where we were previously.

[krzysied]

It looks like it is fixed.