Failing e2e tests on scalability jobs

[shyamjvs]

This is an umbrella issue for tracking currently failing e2e tests on large clusters:

• [sig-instrumentation] Cluster level logging implemented by Stackdriver should ingest system logs from all nodes (Stackdriver logging sink failed to create on large clusters #51700)
• [sig-instrumentation] Monitoring should verify monitoring pods and all cluster nodes are available on influxdb using heapster (Monitoring with influxdb failing on large clusters #51702)
• [sig-network] Firewall rule should have correct firewall rules for e2e cluster (should be fixed by Correct default cluster-ip-range subnet #51343)
• [sig-network] Services should be able to create an internal type load balancer [Slow] (Loadbalancer creation for service failing in case of custom network #51714)
• [sig-storage] Volume plugin streaming [Slow] NFS should write files of various sizes, verify size, validate content (Volume plugin streaming e2e test failing on large cluster #51717)

cc @kubernetes/sig-scalability-misc @wojtek-t @gmarek @spiffxp

[shyamjvs]

Some additional failures seen on 5k-node tests:

• [sig-node] AppArmor load AppArmor profiles should enforce an AppArmor profile (AppArmor profile failing to be enforced on large clusters #51791)
• [sig-scalability] API call latencies shot up on large cluster tests (delete pods API call latencies shot up on large cluster tests #51899)

[k8s-github-robot]

[MILESTONENOTIFIER] Milestone Labels Complete

@shyamjvs

Issue label settings:

• sig/scalability: Issue will be escalated to these SIGs if needed.
• priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
• kind/bug: Fixes a bug discovered during the current release.

Additional instructions available here The commands available for adding these labels are documented here

[dims]

@shyamjvs are we almost done with the issues for this umbrella issue? if so can we close this out and leave the individual issues with correct milestone?

[shyamjvs]

Yes, most of the fixes have been done here - thanks to the concerned SIGs for acting promptly. There are a few ones which are failing even on smaller clusters (like #52496) and are not scalability related. The only one which is failing currently and is actually scalability-related is:

ESIPP [Slow] e2e tests failing on large cluster release-blocking job #52495

There's some discussion going on there about a potential fix but we more or less know why it's failing and don't have to block release on it.
And for failures related to performance tests (which are different from correctness tests), we have separate issues.

[shyamjvs]

Closing the issue. We can reopen it later if needed.

[shyamjvs]

fyi - I've unmarked some of those issues from release-blocker based on the causes identified.