integration framework: log number of leaked goroutines

[aojea]

/kind flake

NONE

Since it will not be easy to avoid the integration framework to leak goroutines due to the complexity of the shutdown process and the way we are setting up the apiserver for testing, at least keep track on how good or bad we are doing

#108060 (comment)

[k8s-ci-robot]

@aojea: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aojea]

/assign @liggitt

[aojea]

/sig testing

[aojea]

example execution

grep "unexpected number of" junit_20220301-103922.stdout
{"Time":"2022-03-01T10:41:57.847915922Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/apiserver/podlogs","Output":"I0301 10:41:57.847865  118737 etcd.go:195] unexpected number of goroutines: before: 3 after 987\n"}
{"Time":"2022-03-01T10:42:01.370413198Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/apiserver/tracing","Output":"I0301 10:42:01.370354  118756 etcd.go:195] unexpected number of goroutines: before: 3 after 1541\n"}
{"Time":"2022-03-01T10:42:45.379251326Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/apimachinery","Output":"I0301 10:42:45.379173  118572 etcd.go:195] unexpected number of goroutines: before: 3 after 2999\n"}
{"Time":"2022-03-01T10:43:39.91644307Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/certificates","Output":"I0301 10:43:39.916398  119063 etcd.go:195] unexpected number of goroutines: before: 3 after 3574\n"}
{"Time":"2022-03-01T10:43:43.939757419Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/apiserver/certreload","Output":"I0301 10:43:43.939397  118710 etcd.go:195] unexpected number of goroutines: before: 3 after 4879\n"}
{"Time":"2022-03-01T10:43:53.616147765Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/configmap","Output":"I0301 10:43:53.616095  119239 etcd.go:195] unexpected number of goroutines: before: 3 after 1007\n"}
{"Time":"2022-03-01T10:44:10.156975602Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/apiserver/flowcontrol","Output":"I0301 10:44:10.156909  118724 etcd.go:195] unexpected number of goroutines: before: 3 after 3101\n"}
{"Time":"2022-03-01T10:44:42.844064347Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/apiserver/admissionwebhook","Output":"I0301 10:44:42.844012  118771 etcd.go:195] unexpected number of goroutines: before: 3 after 3186\n"}
{"Time":"2022-03-01T10:44:55.519286671Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/defaulttolerationseconds","Output":"I0301 10:44:55.519223  120301 etcd.go:195] unexpected number of goroutines: before: 3 after 1007\n"}
{"Time":"2022-03-01T10:45:01.890256903Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/cronjob","Output":"I0301 10:45:01.889938  119387 etcd.go:195] unexpected number of goroutines: before: 3 after 1024\n"}
{"Time":"2022-03-01T10:45:18.738138763Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/auth","Output":"I0301 10:45:18.738087  119005 etcd.go:195] unexpected number of goroutines: before: 3 after 27966\n"}
{"Time":"2022-03-01T10:45:43.365016891Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/dryrun","Output":"I0301 10:45:43.364957  120519 etcd.go:195] unexpected number of goroutines: before: 3 after 246\n"}
{"Time":"2022-03-01T10:45:46.078199165Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/apiserver","Output":"I0301 10:45:46.077881  118907 etcd.go:195] unexpected number of goroutines: before: 3 after 20895\n"}
{"Time":"2022-03-01T10:46:00.661802786Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/apiserver/apply","Output":"I0301 10:46:00.661754  118790 etcd.go:195] unexpected number of goroutines: before: 3 after 39151\n"}
{"Time":"2022-03-01T10:46:06.710605633Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/endpoints","Output":"I0301 10:46:06.710544  120803 etcd.go:195] unexpected number of goroutines: before: 3 after 2009\n"}
{"Time":"2022-03-01T10:46:08.906224147Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/disruption","Output":"I0301 10:46:08.906169  120464 etcd.go:195] unexpected number of goroutines: before: 3 after 1640\n"}
{"Time":"2022-03-01T10:46:21.477826786Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/events","Output":"I0301 10:46:21.477761  120996 etcd.go:195] unexpected number of goroutines: before: 3 after 200\n"}
{"Time":"2022-03-01T10:46:49.806883156Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/evictions","Output":"I0301 10:46:49.806809  121058 etcd.go:195] unexpected number of goroutines: before: 3 after 3012\n"}
{"Time":"2022-03-01T10:46:56.503308106Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/endpointslice","Output":"I0301 10:46:56.502643  120905 etcd.go:195] unexpected number of goroutines: before: 3 after 7010\n"}
{"Time":"2022-03-01T10:47:06.957846369Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/etcd","Output":"I0301 10:47:06.957750  120971 etcd.go:195] unexpected number of goroutines: before: 3 after 4501\n"}
{"Time":"2022-03-01T10:47:10.699271631Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/dualstack","Output":"I0301 10:47:10.691413  120781 etcd.go:195] unexpected number of goroutines: before: 3 after 13975\n"}
{"Time":"2022-03-01T10:47:17.969874628Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/examples","Output":"I0301 10:47:17.967249  121802 etcd.go:195] unexpected number of goroutines: before: 3 after 1317\n"}
{"Time":"2022-03-01T10:47:34.967605207Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/kubelet","Output":"I0301 10:47:34.967560  123056 etcd.go:195] unexpected number of goroutines: before: 3 after 1432\n"}
{"Time":"2022-03-01T10:47:56.054017945Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/metrics","Output":"I0301 10:47:56.053940  123080 etcd.go:195] unexpected number of goroutines: before: 3 after 4991\n"}
{"Time":"2022-03-01T10:47:58.309744199Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/namespace","Output":"I0301 10:47:58.309619  123168 etcd.go:195] unexpected number of goroutines: before: 3 after 2005\n"}
{"Time":"2022-03-01T10:48:04.522640733Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/client","Output":"I0301 10:48:04.522594  119134 etcd.go:195] unexpected number of goroutines: before: 3 after 4268\n"}
{"Time":"2022-03-01T10:48:17.190278945Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/objectmeta","Output":"I0301 10:48:17.190198  123318 etcd.go:195] unexpected number of goroutines: before: 3 after 1007\n"}
{"Time":"2022-03-01T10:48:18.362828725Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/network","Output":"I0301 10:48:18.362768  123248 etcd.go:195] unexpected number of goroutines: before: 3 after 1007\n"}
{"Time":"2022-03-01T10:48:19.810499978Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/deployment","Output":"I0301 10:48:19.810437  120435 etcd.go:195] unexpected number of goroutines: before: 3 after 14241\n"}
{"Time":"2022-03-01T10:48:46.739933673Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/quota","Output":"I0301 10:48:46.739886  123525 etcd.go:195] unexpected number of goroutines: before: 3 after 3029\n"}
{"Time":"2022-03-01T10:48:52.106207756Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/node","Output":"I0301 10:48:52.106162  123281 etcd.go:195] unexpected number of goroutines: before: 3 after 4072\n"}
{"Time":"2022-03-01T10:49:05.008734729Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/scale","Output":"I0301 10:49:05.008678  123649 etcd.go:195] unexpected number of goroutines: before: 3 after 184\n"}
{"Time":"2022-03-01T10:49:07.1047826Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/controlplane","Output":"I0301 10:49:07.104729  119267 etcd.go:195] unexpected number of goroutines: before: 3 after 30252\n"}
{"Time":"2022-03-01T10:49:07.455253754Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/pods","Output":"I0301 10:49:07.454293  123511 etcd.go:195] unexpected number of goroutines: before: 3 after 5987\n"}
{"Time":"2022-03-01T10:49:21.907267029Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/secrets","Output":"I0301 10:49:21.907213  123855 etcd.go:195] unexpected number of goroutines: before: 3 after 1007\n"}
{"Time":"2022-03-01T10:49:53.307871317Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/garbagecollector","Output":"I0301 10:49:53.307792  122651 etcd.go:195] unexpected number of goroutines: before: 3 after 3171\n"}
{"Time":"2022-03-01T10:49:59.514482412Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/job","Output":"I0301 10:49:59.514194  122992 etcd.go:195] unexpected number of goroutines: before: 3 after 21023\n"}
{"Time":"2022-03-01T10:50:01.853002154Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/daemonset","Output":"I0301 10:50:01.852862  120248 etcd.go:195] unexpected number of goroutines: before: 3 after 16076\n"}
{"Time":"2022-03-01T10:50:06.51638754Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/replicaset","Output":"I0301 10:50:06.516067  123548 etcd.go:195] unexpected number of goroutines: before: 3 after 19133\n"}
{"Time":"2022-03-01T10:50:10.551615658Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/service","Output":"I0301 10:50:10.543263  123938 etcd.go:195] unexpected number of goroutines: before: 3 after 9214\n"}
{"Time":"2022-03-01T10:50:11.422042083Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/serviceaccount","Output":"I0301 10:50:11.421993  123977 etcd.go:195] unexpected number of goroutines: before: 3 after 3995\n"}
{"Time":"2022-03-01T10:50:16.603115239Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/replicationcontroller","Output":"I0301 10:50:16.602331  123611 etcd.go:195] unexpected number of goroutines: before: 3 after 14109\n"}
{"Time":"2022-03-01T10:50:37.414567406Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/storageclasses","Output":"I0301 10:50:37.414509  126114 etcd.go:195] unexpected number of goroutines: before: 3 after 1007\n"}
{"Time":"2022-03-01T10:50:39.969099605Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/tls","Output":"I0301 10:50:39.969017  126146 etcd.go:195] unexpected number of goroutines: before: 3 after 243\n"}
{"Time":"2022-03-01T10:50:44.863611991Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/ttlcontroller","Output":"I0301 10:50:44.863555  126156 etcd.go:195] unexpected number of goroutines: before: 3 after 1007\n"}
{"Time":"2022-03-01T10:51:03.264159671Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/statefulset","Output":"I0301 10:51:03.264099  126122 etcd.go:195] unexpected number of goroutines: before: 3 after 4271\n"}
{"Time":"2022-03-01T10:51:04.137319693Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/storageversion","Output":"I0301 10:51:04.137264  126129 etcd.go:195] unexpected number of goroutines: before: 3 after 539\n"}
{"Time":"2022-03-01T10:51:13.455637634Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/serving","Output":"I0301 10:51:13.455593  126101 etcd.go:195] unexpected number of goroutines: before: 3 after 799\n"}
{"Time":"2022-03-01T10:52:38.327731961Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/volume","Output":"I0301 10:52:38.327669  126211 etcd.go:195] unexpected number of goroutines: before: 3 after 15019\n"}
{"Time":"2022-03-01T10:54:34.048323524Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/volumescheduling","Output":"I0301 10:54:34.048239  126311 etcd.go:195] unexpected number of goroutines: before: 3 after 14290\n"}
{"Time":"2022-03-01T10:57:23.341487254Z","Action":"output","Package":"k8s.io/kubernetes/test/integration/scheduler","Output":"I0301 10:57:23.341382  123806 etcd.go:195] unexpected number of goroutines: before: 3 after 68556\n"}

[liggitt]

one request for a clarifying comment, then lgtm

[aojea]

/test pull-kubernetes-conformance-kind-ipv6-parallel

[liggitt]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/OWNERS [aojea,liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment