Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flaking Test] integration-master (goroutine leak detection) #125028

Closed
Vyom-Yadav opened this issue May 21, 2024 · 12 comments · Fixed by #125739
Closed

[Flaking Test] integration-master (goroutine leak detection) #125028

Vyom-Yadav opened this issue May 21, 2024 · 12 comments · Fixed by #125739
Assignees
@seans3 @lowang-bh
Labels
kind/flake Categorizes issue or PR as related to a flaky test. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/testing Categorizes an issue or PR as relevant to SIG Testing. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@Vyom-Yadav
Copy link
Member

Vyom-Yadav commented May 21, 2024 •
edited
Loading

Which jobs are flaking?

master-blocking:

  • integration-master

Which tests are flaking?

  • k8s.io/kubernetes/test/integration/kubelet.kubelet
  • k8s.io/kubernetes/test/integration/apiserver: portforward

Since when has it been flaking?

This has been occasionally flaking for some time. Recent Failures:

4/30/2024, 1:42:50 PM ci-kubernetes-integration-1-30
4/29/2024, 6:33:48 PM ci-kubernetes-integration-master
4/22/2024, 1:05:57 PM ci-kubernetes-integration-1-30

2024-06-03 22:44:28 +0000 UTC

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#integration-master

Reason for failure (if possible)

k8s.io/kubernetes/test/integration/kubelet.kubelet

E0429 13:44:12.935549  115466 etcd.go:221] "EtcdMain goroutine check" err=<
	found unexpected goroutines:
	[Goroutine 29225 in state select, with golang.org/x/net/http2.(*serverConn).serve on top of the stack:
	golang.org/x/net/http2.(*serverConn).serve(0xc006b7ba00)

k8s.io/kubernetes/test/integration/apiserver: portforward

E0603 23:09:41.143906  109550 etcd.go:221] "EtcdMain goroutine check" err=<
	found unexpected goroutines:
	[Goroutine 4444 in state chan send, with github.com/moby/spdystream.(*Connection).shutdown on top of the stack:
	github.com/moby/spdystream.(*Connection).shutdown(0xc0042c4f20, 0x0)
		/home/prow/go/src/k8s.io/kubernetes/vendor/github.com/moby/spdystream/connection.go:743 +0x285
	created by github.com/moby/spdystream.(*Connection).Close in goroutine 4437
		/home/prow/go/src/k8s.io/kubernetes/vendor/github.com/moby/spdystream/connection.go:769 +0x145
	]
 >
FAIL	k8s.io/kubernetes/test/integration/apiserver/portforward	9.811s

Anything else we need to know?

Another leak was fixed at #116196.

Relevant SIG(s)

/sig api-machinery
/sig testing
@Vyom-Yadav Vyom-Yadav added the kind/flake Categorizes issue or PR as related to a flaky test. label May 21, 2024
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/testing Categorizes an issue or PR as relevant to SIG Testing. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 21, 2024
@Vyom-Yadav
Copy link
Member Author

/retitle [Flaking Test] integration-master (goroutine leak detection)

@k8s-ci-robot k8s-ci-robot changed the title [Falking Test] integration-master (goroutine leak detection) [Flaking Test] integration-master (goroutine leak detection) May 21, 2024
@aojea
Copy link
Member

aojea commented May 21, 2024

Some server is not stopping gracefully on that test

=== FAIL: test/integration/kubelet  (0.00s)
I0521 08:57:09.471381  113893 etcd.go:71] etcd already running at http://127.0.0.1:2379/
PASS
E0521 08:58:19.969370  113893 etcd.go:221] "EtcdMain goroutine check" err=<
	found unexpected goroutines:
	[Goroutine 29874 in state select, 1 minutes, with golang.org/x/net/http2.(*serverConn).serve on top of the stack:
	golang.org/x/net/http2.(*serverConn).serve(0xc00d2c4000)
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:941 +0x897
	golang.org/x/net/http2.(*Server).ServeConn(0xc00382a5f0, {0x3a38a50, 0xc00d071c08}, 0xc006733b30)
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:532 +0xbad
	golang.org/x/net/http2.ConfigureServer.func1(0xc002a55c20, 0xc00d071c08, {0x39f5420, 0xc00c8470c0})
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:322 +0x125
	net/http.(*conn).serve(0xc00c51ac60, {0x3a29ed0, 0xc00482f7d0})
		/usr/local/go/src/net/http/server.go:1947 +0x12f3
	created by net/http.(*Server).Serve in goroutine 2468
		/usr/local/go/src/net/http/server.go:3285 +0x4b4
	 Goroutine 29866 in state IO wait, 1 minutes, with internal/poll.runtime_pollWait on top of the stack:
	internal/poll.runtime_pollWait(0x7f6bb04788a8, 0x72)
		/usr/local/go/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc00b41f700?, 0xc005332000?, 0x0)
		/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc00b41f700, {0xc005332000, 0xa80, 0xa80})
		/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc00b41f700, {0xc005332000?, 0x7f6bb033de88?, 0xc00b36e978?})
		/usr/local/go/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc004791250, {0xc005332000?, 0xc00a5d1938?, 0x41467b?})
		/usr/local/go/src/net/net.go:179 +0x45
	crypto/tls.(*atLeastReader).Read(0xc00b36e978, {0xc005332000?, 0x0?, 0xc00b36e978?})
		/usr/local/go/src/crypto/tls/conn.go:806 +0x3b
	bytes.(*Buffer).ReadFrom(0xc00cb65430, {0x39f7500, 0xc00b36e978})
		/usr/local/go/src/bytes/buffer.go:211 +0x98
	crypto/tls.(*Conn).readFromUntil(0xc00cb65188, {0x39f4cc0, 0xc004791250}, 0xc00a5d1980?)
		/usr/local/go/src/crypto/tls/conn.go:828 +0xde
	crypto/tls.(*Conn).readRecordOrCCS(0xc00cb65188, 0x0)
		/usr/local/go/src/crypto/tls/conn.go:626 +0x3cf
	crypto/tls.(*Conn).readRecord(...)
		/usr/local/go/src/crypto/tls/conn.go:588
	crypto/tls.(*Conn).Read(0xc00cb65188, {0xc006601000, 0x1000, 0x4481a9?})
		/usr/local/go/src/crypto/tls/conn.go:1370 +0x156
	bufio.(*Reader).Read(0xc005330de0, {0xc0062b2f20, 0x9, 0x46243e?})
		/usr/local/go/src/bufio/bufio.go:241 +0x197
	io.ReadAtLeast({0x39f3f40, 0xc005330de0}, {0xc0062b2f20, 0x9, 0x9}, 0x9)
		/usr/local/go/src/io/io.go:335 +0x90
	io.ReadFull(...)
		/usr/local/go/src/io/io.go:354
	golang.org/x/net/http2.readFrameHeader({0xc0062b2f20, 0x9, 0x9c9a05?}, {0x39f3f40?, 0xc005330de0?})
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:237 +0x65
	golang.org/x/net/http2.(*Framer).ReadFrame(0xc0062b2ee0)
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:498 +0x85
	golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00a5d1fa8)
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:2429 +0xd8
	golang.org/x/net/http2.(*ClientConn).readLoop(0xc00e3c4d80)
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:2325 +0x65
	created by golang.org/x/net/http2.(*ClientConn).goRun in goroutine 29865
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:369 +0x2d
	 Goroutine 29853 in state IO wait, 1 minutes, with internal/poll.runtime_pollWait on top of the stack:
	internal/poll.runtime_pollWait(0x7f6bb3242700, 0x72)
		/usr/local/go/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc00b704480?, 0xc00d19efc0?, 0x0)
		/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc00b704480, {0xc00d19efc0, 0x240, 0x240})
		/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc00b704480, {0xc00d19efc0?, 0x7f6bb033de88?, 0xc00b36e960?})
		/usr/local/go/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc000511b90, {0xc00d19efc0?, 0xc00a3aea58?, 0x41467b?})
		/usr/local/go/src/net/net.go:179 +0x45
	crypto/tls.(*atLeastReader).Read(0xc00b36e960, {0xc00d19efc0?, 0x0?, 0xc00b36e960?})
		/usr/local/go/src/crypto/tls/conn.go:806 +0x3b
	bytes.(*Buffer).ReadFrom(0xc00d071eb0, {0x39f7500, 0xc00b36e960})
		/usr/local/go/src/bytes/buffer.go:211 +0x98
	crypto/tls.(*Conn).readFromUntil(0xc00d071c08, {0x39f4cc0, 0xc000511b90}, 0xc00a3aeaa0?)
		/usr/local/go/src/crypto/tls/conn.go:828 +0xde
	crypto/tls.(*Conn).readRecordOrCCS(0xc00d071c08, 0x0)
		/usr/local/go/src/crypto/tls/conn.go:626 +0x3cf
	crypto/tls.(*Conn).readRecord(...)
		/usr/local/go/src/crypto/tls/conn.go:588
	crypto/tls.(*Conn).Read(0xc00d071c08, {0xc00d2ba200, 0x9, 0x455bc6?})
		/usr/local/go/src/crypto/tls/conn.go:1370 +0x156
	io.ReadAtLeast({0x7f6bb0477f58, 0xc00d071c08}, {0xc00d2ba200, 0x9, 0x9}, 0x9)
		/usr/local/go/src/io/io.go:335 +0x90
	io.ReadFull(...)
		/usr/local/go/src/io/io.go:354
	golang.org/x/net/http2.readFrameHeader({0xc00d2ba200, 0x9, 0x0?}, {0x7f6bb0477f58?, 0xc00d071c08?})
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:237 +0x65
	golang.org/x/net/http2.(*Framer).ReadFrame(0xc00d2ba1c0)
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:498 +0x85
	golang.org/x/net/http2.(*serverConn).readFrames(0xc00d2c4000)
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:821 +0x87
	created by golang.org/x/net/http2.(*serverConn).serve in goroutine 29874
		/home/prow/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:933 +0x56a
	]
 >
FAIL	k8s.io/kubernetes/test/integration/kubelet	70.654s

@MadhavJivrajani
Copy link
Contributor

Similar reason: #122051

@MadhavJivrajani
Copy link
Contributor

Its either not terminating gracefully or there is a request that isn't being being served in a timely manner, because I also see a bunch of goroutines waiting on the netpoller.

@seans3
Copy link
Contributor

seans3 commented May 21, 2024

/triage accepted (for SIG api-machinery)

@k8s-ci-robot k8s-ci-robot added the triage/accepted Indicates an issue or PR is ready to be actively worked on. label May 21, 2024
@k8s-ci-robot
Copy link
Contributor

@seans3: The label(s) triage/(for, triage/sig, triage/api-machinery) cannot be applied, because the repository doesn't have them.

In response to this:

/triage accepted (for SIG api-machinery)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot removed the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label May 21, 2024
@Vyom-Yadav
Copy link
Member Author

@seans3 @aojea @MadhavJivrajani I don't think this would be a release blocker, still can you confirm whether this would block the upcoming alpha 1 release cut?

@Vyom-Yadav
Copy link
Member Author

Another failure:
k8s.io/kubernetes/test/integration/apiserver: portforward

E0603 23:09:41.143906  109550 etcd.go:221] "EtcdMain goroutine check" err=<
	found unexpected goroutines:
	[Goroutine 4444 in state chan send, with github.com/moby/spdystream.(*Connection).shutdown on top of the stack:
	github.com/moby/spdystream.(*Connection).shutdown(0xc0042c4f20, 0x0)
		/home/prow/go/src/k8s.io/kubernetes/vendor/github.com/moby/spdystream/connection.go:743 +0x285
	created by github.com/moby/spdystream.(*Connection).Close in goroutine 4437
		/home/prow/go/src/k8s.io/kubernetes/vendor/github.com/moby/spdystream/connection.go:769 +0x145
	]

@wendy-ha18
Copy link
Member

Hey @deads2k , @jpbetz
The deadline for the alpha 1 release cut has been delayed to Monday, June 10 and we are still waiting for confirmation of this issue. Do you think this will be a release blocker?

@lowang-bh
Copy link
Member

/assign

@seans3
Copy link
Contributor

seans3 commented Jun 17, 2024

/assign

@seans3
Copy link
Contributor

seans3 commented Jun 26, 2024

...for portforward integration test flake, we believe this PR fixes it: #125739.

I'm not marking this fixed because it also mentions a kubelet integration test flake.

@seans3 seans3 mentioned this issue Jun 26, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@seans3 seans3

@lowang-bh lowang-bh

Labels
kind/flake Categorizes issue or PR as related to a flaky test. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/testing Categorizes an issue or PR as relevant to SIG Testing. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
CI Signal (SIG Release / Release Team)
Status: RESOLVED
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ignore reported goroutine leak during SPDY shutdown seans3/kubernetes
7 participants
@aojea @seans3 @MadhavJivrajani @k8s-ci-robot @lowang-bh @Vyom-Yadav @wendy-ha18