Suspicious breakdown in pod startup time in scalability tests

[wojtek-t]

I've already observed a number cases in scalability tests, where for pod startup-time, the breakdown looks suspicious.
As an example, in this run:
https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/255/build-log.txt

from looking into 10% worst times, we see that:

I1113 02:27:31.355] Nov 13 02:27:31.301: INFO: 10% worst schedule-to-watch latencies: ...  {density-latency-pod-438-8thtq gce-scale-cluster-minion-group-2-4km0 4.881956298s}]
...
I1113 02:27:31.376] Nov 13 02:27:31.324: INFO: 10% worst e2e latencies: ... {density-latency-pod-438-8thtq gce-scale-cluster-minion-group-2-4km0 4.881956298s}]

The time for "schedule-to-watch" and "e2e latencies" for different pods are exactly the same (which suggests something is wrong).

The second thing that is a bit suspicious in some cases is that the "watch part" something is relatively long (time from when kubelet reports pod status as running to when test really observes that, even though watch latencies e.g. for scheduler or controller-manager are low). This may or may not be related to some starvation at the test level.

@kubernetes/sig-scalability-bugs @mborsz

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[wojtek-t]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[wojtek-t]

/lifecycle frozen