PR builder failing with: Quota 'NETWORKS' exceeded. Limit: 75.0

[zmerlynn]

From http://kubekins.dls.corp.google.com:8081/job/kubernetes-pull-build-test-e2e-gce/41958/:

+++ Staging server tars to Google Storage: gs://kubernetes-staging-b6e85ca3f3/e2e-gce-agent-pr-5-2-devel
+++ kubernetes-server-linux-amd64.tar.gz uploaded (sha1 = a9d0980fd8e8964b93de951441da59eb23274b3f)
+++ kubernetes-salt.tar.gz uploaded (sha1 = c6ad0892e67923780b9ed9ac687799a14525082f)
+++ kubernetes-manifests.tar.gz uploaded (sha1 = 31078c562df80a6db7e20e97ec61f975a4c002db)
Creating new network: e2e-gce-agent-pr-5-2
WARNING: You are creating a legacy network. Using --mode=legacy will be required in future releases.
ERROR: (gcloud.compute.networks.create) Some requests did not succeed:
 - Quota 'NETWORKS' exceeded. Limit: 75.0

[zmerlynn]

Also here: http://kubekins.dls.corp.google.com:8081/job/kubernetes-pull-build-test-e2e-gce/41956/console, from #26263

[dims]

@zmerlynn - Are there public logs?

[zmerlynn]

Here's one set.

[euank]

Another set of logs: https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/25902/kubernetes-pull-build-test-e2e-gce/41986/ from #25902

ERROR: (gcloud.compute.networks.create) Some requests did not succeed:
 - Quota 'NETWORKS' exceeded. Limit: 75.0

[yujuhong]

https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/26272/kubernetes-pull-build-test-e2e-gce/41981/

[nikhiljindal]

cc @kubernetes/test-infra-maintainers

[lavalamp]

@ixdy @fejta @spxtr Too many simultaneous runs in one project?

[alex-mohr]

I chatted with @rmmh who offered to make sure someone on @kubernetes/test-infra-maintainers works on it.

[alex-mohr]

I wonder if this is a symptom of #26199 fixed by #26208 ? Presumably have to clean up too even if PR fixes future leaks.

[ixdy]

The issue is that we create networks and routes based on the Jenkins agent name, but these never get cleaned up by the cluster scripts. @fejta recently renamed all of the Jenkins agents, which resulted in us creating a bunch of new networks and not deleting the old deprecated names.

I'll clean up the old ones now.

[ixdy]

gcloud compute firewall-rules list --project=kubernetes-jenkins-pull --format='value(name)' | grep builder-'[12345]' | xargs gcloud compute firewall-rules delete --project=kubernetes-jenkins-pull
gcloud compute networks list --project=kubernetes-jenkins-pull --format='value(name)' | grep builder-'[12345]'  | xargs gcloud compute networks delete --project=kubernetes-jenkins-pull

Back under quota limits.