New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubelet leaks memory on kubectl port-forward #57992

Closed
SamClinckspoor opened this Issue Jan 9, 2018 · 11 comments

Comments

Projects
None yet
6 participants
@SamClinckspoor
Copy link

SamClinckspoor commented Jan 9, 2018

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:
When opening port-forward requests to a pod, kubelet memory usage increases without releasing it again when the port-forward is stopped.

What you expected to happen:
Kubelet should cleanup the port-forward when it is closed

How to reproduce it (as minimally and precisely as possible):
Using the following script I can observe an increase in Kubelet memory usage on the node where the pod runs:

for i in {8000..8300}; do
kubectl port-forward anypod $i:8080 &; sleep 0.2;
done

...
killall kubectl

300 connections allocates roughly 30mb by kubelet

We first noticed this on the node where Tiller (Helm) runs. The node regularly runs out of memory because of the Kubelet memory usage. see also helm/helm#3121 We don't experience the helm issue on all clusters, but I can reproduce the memory leak with the above script on all clusters.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T19:59:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.11", GitCommit:"b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55", GitTreeState:"clean", BuildDate:"2017-11-25T17:51:39Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release): Debian jessie
  • Kernel (e.g. uname -a): Linux 4.4.102-k8s #1 SMP Sun Nov 26 23:32:43 UTC 2017 x86_64 GNU/Linux
  • Install tools: Kops 1.8.0
  • Others:
@anguslees

This comment has been minimized.

Copy link
Member

anguslees commented Jan 9, 2018

/sig node

In my testing (v1.8.0), I observed kubectl port-forward create a TCP connection between apiserver and kubelet for each port-forward (and one between apiserver and kubectl). This apiserver<->kubelet TCP connection was not torn down when the corresponding kubectl port-forward command was killed.

I have not tried against a newer apiserver/kubelet, but if someone has one handy, it's easy to just compare ss -t output on the apiserver or kubelet host before and after running several kubectl port-forward commands. Be aware that the CLOSE_WAIT connections can take minutes to expire after the kubectl<->apiserver commands exit, and this is normal and fine - we're interested in the ones that stay in state ESTABLISHED.

@k8s-ci-robot k8s-ci-robot added sig/node and removed needs-sig labels Jan 9, 2018

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Apr 4, 2018

I'm not aware of any changes in 1.9 or 1.10 that would have fixed this.

Not that we shouldn't fix this issue, but something like https://github.com/kubernetes/kubernetes/pull/12498/files would be a good backstop to leaking connections like this

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jul 3, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@SamClinckspoor

This comment has been minimized.

Copy link

SamClinckspoor commented Jul 3, 2018

/remove-lifecycle stale

behaviour is still observable for us on 1.9

@MHBauer

This comment has been minimized.

Copy link
Member

MHBauer commented Jul 31, 2018

On v1.11 I see plenty of CLOSE_WAIT and ESTABLISHED connections that hang around after many minutes.

@MHBauer

This comment has been minimized.

Copy link
Member

MHBauer commented Aug 1, 2018

I added some logging to (among many other places)
https://github.com/kubernetes/kubernetes/blob/v1.11.1/staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go#L297-L329
After closing a port-forward from kubectl, I see immediately the reader close on the apiserver, and the writer close on the kubelet.
I do not see the other side of the connections close. Maybe 1 on each side when the pod ends.
Is there a flag to enable pprof so we can dump some goroutine stacktraces?
/sig apimachinery

@MHBauer

This comment has been minimized.

Copy link
Member

MHBauer commented Aug 1, 2018

kubelet stacktrace, the important parts

goroutine profile: total 3565
1010 @ 0x42fcaa 0x42fd5e 0x407322 0x406fdb 0x16cb228 0x45d591
#	0x16cb227	k8s.io/kubernetes/vendor/github.com/docker/spdystream.(*Connection).Serve.func1+0x37	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/spdystream/connection.go:306

1010 @ 0x42fcaa 0x42fd5e 0x44176b 0x471420 0x16c9796 0x16c660f 0x16cb2b5 0x45d591
#	0x44176a	sync.runtime_notifyListWait+0x10a							/usr/local/go/src/runtime/sema.go:510
#	0x47141f	sync.(*Cond).Wait+0x7f									/usr/local/go/src/sync/cond.go:56
#	0x16c9795	k8s.io/kubernetes/vendor/github.com/docker/spdystream.(*PriorityFrameQueue).Pop+0x85	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/spdystream/priority.go:86
#	0x16c660e	k8s.io/kubernetes/vendor/github.com/docker/spdystream.(*Connection).frameHandler+0x2e	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/spdystream/connection.go:398
#	0x16cb2b4	k8s.io/kubernetes/vendor/github.com/docker/spdystream.(*Connection).Serve.func2+0x64	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/spdystream/connection.go:315

305 @ 0x42fcaa 0x42afca 0x42a647 0x49929b 0x49931d 0x49a17d 0x511ccf 0x52413a 0x647c06 0x648110 0x64b6b0 0x474e44 0x474baa 0x525cd4 0x52d785 0x52b3bf 0x474ffa 0x474baa 0x16dc7bc 0x45d591
#	0x42a646	internal/poll.runtime_pollWait+0x56										/usr/local/go/src/runtime/netpoll.go:173
#	0x49929a	internal/poll.(*pollDesc).wait+0x9a										/usr/local/go/src/internal/poll/fd_poll_runtime.go:85
#	0x49931c	internal/poll.(*pollDesc).waitRead+0x3c										/usr/local/go/src/internal/poll/fd_poll_runtime.go:90
#	0x49a17c	internal/poll.(*FD).Read+0x17c											/usr/local/go/src/internal/poll/fd_unix.go:157
#	0x511cce	net.(*netFD).Read+0x4e												/usr/local/go/src/net/fd_unix.go:202
#	0x524139	net.(*conn).Read+0x69												/usr/local/go/src/net/net.go:176
#	0x647c05	crypto/tls.(*block).readFromUntil+0x95										/usr/local/go/src/crypto/tls/conn.go:493
#	0x64810f	crypto/tls.(*Conn).readRecord+0xdf										/usr/local/go/src/crypto/tls/conn.go:595
#	0x64b6af	crypto/tls.(*Conn).Read+0xff											/usr/local/go/src/crypto/tls/conn.go:1156
#	0x474e43	io.copyBuffer+0x163												/usr/local/go/src/io/io.go:400
#	0x474ba9	io.Copy+0x59													/usr/local/go/src/io/io.go:362
#	0x525cd3	net.genericReadFrom+0x83											/usr/local/go/src/net/net.go:597
#	0x52d784	net.(*TCPConn).readFrom+0xa4											/usr/local/go/src/net/tcpsock_posix.go:51
#	0x52b3be	net.(*TCPConn).ReadFrom+0x5e											/usr/local/go/src/net/tcpsock.go:103
#	0x474ff9	io.copyBuffer+0x319												/usr/local/go/src/io/io.go:386
#	0x474ba9	io.Copy+0x59													/usr/local/go/src/io/io.go:362
#	0x16dc7bb	k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).tryUpgrade.func1+0x11b	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:312

305 @ 0x42fcaa 0x42fd5e 0x440644 0x440269 0x4743d2 0x16db530 0x16da4c3 0x369ab30 0x369bc76 0x369da3e 0x77c7fe 0x7725be 0x369c75b 0x772598 0x76f28b 0x77d2a8 0x6ef724 0x6f1490 0x76f75d 0x369bea4 0x6f253c 0x6ee741 0x45d591
#	0x440268	sync.runtime_Semacquire+0x38															/usr/local/go/src/runtime/sema.go:56
#	0x4743d1	sync.(*WaitGroup).Wait+0x71															/usr/local/go/src/sync/waitgroup.go:129
#	0x16db52f	k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).tryUpgrade+0x8cf						/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:336
#	0x16da4c2	k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).ServeHTTP+0xb2						/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:187
#	0x369ab2f	k8s.io/kubernetes/pkg/kubelet/server.proxyStream+0x8f												/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:638
#	0x369bc75	k8s.io/kubernetes/pkg/kubelet/server.(*Server).getPortForward+0x555										/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:764
#	0x369da3d	k8s.io/kubernetes/pkg/kubelet/server.(*Server).(k8s.io/kubernetes/pkg/kubelet/server.getPortForward)-fm+0x3d					/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:353
#	0x77c7fd	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch.func5+0x3d							/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272
#	0x7725bd	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*FilterChain).ProcessFilter+0x8d							/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/filter.go:21
#	0x369c75a	k8s.io/kubernetes/pkg/kubelet/server.(*Server).InstallAuthFilter.func1+0x4da									/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:250
#	0x772597	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*FilterChain).ProcessFilter+0x67							/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/filter.go:19
#	0x76f28a	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch+0x7ea								/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:274
#	0x77d2a7	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm+0x47	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120
#	0x6ef723	net/http.HandlerFunc.ServeHTTP+0x43														/usr/local/go/src/net/http/server.go:1947
#	0x6f148f	net/http.(*ServeMux).ServeHTTP+0x12f														/usr/local/go/src/net/http/server.go:2337
#	0x76f75c	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP+0x4c								/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:292
#	0x369bea3	k8s.io/kubernetes/pkg/kubelet/server.(*Server).ServeHTTP+0x103											/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:780
#	0x6f253b	net/http.serverHandler.ServeHTTP+0xbb														/usr/local/go/src/net/http/server.go:2694
#	0x6ee740	net/http.(*conn).serve+0x650															/usr/local/go/src/net/http/server.go:1830

202 @ 0x42fcaa 0x42afca 0x42a647 0x49929b 0x49931d 0x49a17d 0x511ccf 0x52413a 0x474e44 0x474baa 0x16dca5b 0x45d591
#	0x42a646	internal/poll.runtime_pollWait+0x56										/usr/local/go/src/runtime/netpoll.go:173
#	0x49929a	internal/poll.(*pollDesc).wait+0x9a										/usr/local/go/src/internal/poll/fd_poll_runtime.go:85
#	0x49931c	internal/poll.(*pollDesc).waitRead+0x3c										/usr/local/go/src/internal/poll/fd_poll_runtime.go:90
#	0x49a17c	internal/poll.(*FD).Read+0x17c											/usr/local/go/src/internal/poll/fd_unix.go:157
#	0x511cce	net.(*netFD).Read+0x4e												/usr/local/go/src/net/fd_unix.go:202
#	0x524139	net.(*conn).Read+0x69												/usr/local/go/src/net/net.go:176
#	0x474e43	io.copyBuffer+0x163												/usr/local/go/src/io/io.go:400
#	0x474ba9	io.Copy+0x59													/usr/local/go/src/io/io.go:362
#	0x16dca5a	k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).tryUpgrade.func2+0x13a	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:327

202 @ 0x42fcaa 0x42afca 0x42a647 0x49929b 0x49931d 0x49a17d 0x511ccf 0x52413a 0x6e9628 0x544b28 0x16ce4b4 0x474886 0x4749f8 0x5a4275 0x16bf7bf 0x16c5072 0x16c5cb5 0x45d591
#	0x42a646	internal/poll.runtime_pollWait+0x56								/usr/local/go/src/runtime/netpoll.go:173
#	0x49929a	internal/poll.(*pollDesc).wait+0x9a								/usr/local/go/src/internal/poll/fd_poll_runtime.go:85
#	0x49931c	internal/poll.(*pollDesc).waitRead+0x3c								/usr/local/go/src/internal/poll/fd_poll_runtime.go:90
#	0x49a17c	internal/poll.(*FD).Read+0x17c									/usr/local/go/src/internal/poll/fd_unix.go:157
#	0x511cce	net.(*netFD).Read+0x4e										/usr/local/go/src/net/fd_unix.go:202
#	0x524139	net.(*conn).Read+0x69										/usr/local/go/src/net/net.go:176
#	0x6e9627	net/http.(*connReader).Read+0xf7								/usr/local/go/src/net/http/server.go:764
#	0x544b27	bufio.(*Reader).Read+0x237									/usr/local/go/src/bufio/bufio.go:216
#	0x16ce4b3	k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy.(*connWrapper).Read+0x83	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy/upgrade.go:54
#	0x474885	io.ReadAtLeast+0x85										/usr/local/go/src/io/io.go:309
#	0x4749f7	io.ReadFull+0x57										/usr/local/go/src/io/io.go:327
#	0x5a4274	encoding/binary.Read+0xe4									/usr/local/go/src/encoding/binary/binary.go:171
#	0x16bf7be	k8s.io/kubernetes/vendor/github.com/docker/spdystream/spdy.(*Framer).ReadFrame+0x7e		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/spdystream/spdy/read.go:148
#	0x16c5071	k8s.io/kubernetes/vendor/github.com/docker/spdystream.(*idleAwareFramer).ReadFrame+0x31		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/spdystream/connection.go:151
#	0x16c5cb4	k8s.io/kubernetes/vendor/github.com/docker/spdystream.(*Connection).Serve+0x174			/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/spdystream/connection.go:325

202 @ 0x42fcaa 0x43f9c0 0x16c4ba1 0x45d591
#	0x16c4ba0	k8s.io/kubernetes/vendor/github.com/docker/spdystream.(*idleAwareFramer).monitor+0x170	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/spdystream/connection.go:61

202 @ 0x42fcaa 0x43f9c0 0x34efd9f 0x34eee9b 0x34f1026 0x36946a9 0x3694ebe 0x76f459 0x77d2a8 0x6ef724 0x6f1490 0x76f75d 0x6f253c 0x6ee741 0x45d591
#	0x34efd9e	k8s.io/kubernetes/pkg/kubelet/server/portforward.(*httpStreamHandler).run+0x19e									/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/portforward/httpstream.go:212
#	0x34eee9a	k8s.io/kubernetes/pkg/kubelet/server/portforward.handleHttpStreams+0x37a									/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/portforward/httpstream.go:65
#	0x34f1025	k8s.io/kubernetes/pkg/kubelet/server/portforward.ServePortForward+0x2a5										/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/portforward/portforward.go:49
#	0x36946a8	k8s.io/kubernetes/pkg/kubelet/server/streaming.(*server).servePortForward+0x1c8									/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/streaming/server.go:352
#	0x3694ebd	k8s.io/kubernetes/pkg/kubelet/server/streaming.(*server).(k8s.io/kubernetes/pkg/kubelet/server/streaming.servePortForward)-fm+0x3d		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/streaming/server.go:132
#	0x76f458	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch+0x9b8								/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277
#	0x77d2a7	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm+0x47	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120
#	0x6ef723	net/http.HandlerFunc.ServeHTTP+0x43														/usr/local/go/src/net/http/server.go:1947
#	0x6f148f	net/http.(*ServeMux).ServeHTTP+0x12f														/usr/local/go/src/net/http/server.go:2337
#	0x76f75c	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP+0x4c								/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:292
#	0x6f253b	net/http.serverHandler.ServeHTTP+0xbb														/usr/local/go/src/net/http/server.go:2694
#	0x6ee740	net/http.(*conn).serve+0x650															/usr/local/go/src/net/http/server.go:1830
@MHBauer

This comment has been minimized.

Copy link
Member

MHBauer commented Aug 1, 2018

/sig api-machinery

@MHBauer

This comment has been minimized.

Copy link
Member

MHBauer commented Aug 1, 2018

apiserver stacktrace, the important parts

goroutine profile: total 2470
308 @ 0x42fcaa 0x42afca 0x42a647 0x49929b 0x49931d 0x49a17d 0x511ccf 0x52413a 0x647c06 0x648110 0x64b6b0 0x474e44 0x474baa 0x525cd4 0x52d785 0x52b3bf 0x474ffa 0x474baa 0x16dca5b 0x45d591
#	0x42a646	internal/poll.runtime_pollWait+0x56										/usr/local/go/src/runtime/netpoll.go:173
#	0x49929a	internal/poll.(*pollDesc).wait+0x9a										/usr/local/go/src/internal/poll/fd_poll_runtime.go:85
#	0x49931c	internal/poll.(*pollDesc).waitRead+0x3c										/usr/local/go/src/internal/poll/fd_poll_runtime.go:90
#	0x49a17c	internal/poll.(*FD).Read+0x17c											/usr/local/go/src/internal/poll/fd_unix.go:157
#	0x511cce	net.(*netFD).Read+0x4e												/usr/local/go/src/net/fd_unix.go:202
#	0x524139	net.(*conn).Read+0x69												/usr/local/go/src/net/net.go:176
#	0x647c05	crypto/tls.(*block).readFromUntil+0x95										/usr/local/go/src/crypto/tls/conn.go:493
#	0x64810f	crypto/tls.(*Conn).readRecord+0xdf										/usr/local/go/src/crypto/tls/conn.go:595
#	0x64b6af	crypto/tls.(*Conn).Read+0xff											/usr/local/go/src/crypto/tls/conn.go:1156
#	0x474e43	io.copyBuffer+0x163												/usr/local/go/src/io/io.go:400
#	0x474ba9	io.Copy+0x59													/usr/local/go/src/io/io.go:362
#	0x525cd3	net.genericReadFrom+0x83											/usr/local/go/src/net/net.go:597
#	0x52d784	net.(*TCPConn).readFrom+0xa4											/usr/local/go/src/net/tcpsock_posix.go:51
#	0x52b3be	net.(*TCPConn).ReadFrom+0x5e											/usr/local/go/src/net/tcpsock.go:103
#	0x474ff9	io.copyBuffer+0x319												/usr/local/go/src/io/io.go:386
#	0x474ba9	io.Copy+0x59													/usr/local/go/src/io/io.go:362
#	0x16dca5a	k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).tryUpgrade.func2+0x13a	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:327

308 @ 0x42fcaa 0x42fd5e 0x440644 0x440269 0x4743d2 0x16db530 0x16da4c3 0x144473d 0xfa5855 0x1444b4f 0x145e5b6 0xfa7238 0x76f459 0x76ea47 0x14847dc 0x1489a55 0x16e39b9 0x1270436 0x127018a 0x1484998 0x1489a55 0xfb9ee3 0x6ef724 0x1269d0c 0x126bbd7 0x6ef724 0x126c244 0x6ef724 0xfbd6a1 0x6ef724 0x126c658 0x6ef724
#	0x440268	sync.runtime_Semacquire+0x38										/usr/local/go/src/runtime/sema.go:56
#	0x4743d1	sync.(*WaitGroup).Wait+0x71										/usr/local/go/src/sync/waitgroup.go:129
#	0x16db52f	k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).tryUpgrade+0x8cf	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:336
#	0x16da4c2	k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).ServeHTTP+0xb2	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:187
#	0x144473c	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ConnectResource.func1.1+0x24c		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:145
#	0xfa5854	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.RecordLongRunning+0x1c4			/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:188
#	0x1444b4e	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ConnectResource.func1+0x38e		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:139
#	0x145e5b5	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulConnectResource.func1+0xd5		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1062
#	0xfa7237	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1+0x207		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:225
#	0x76f458	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch+0x9b8			/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277
#	0x76ea46	k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).Dispatch+0x56			/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199
#	0x14847db	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP+0x4eb				/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:146
#	0x16e39b8	k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP+0x168		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91
#	0x1270435	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP+0x265			/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248
#	0x1270189	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP+0x89		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234
#	0x1484997	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP+0x6a7				/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154
#	0xfb9ee2	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1+0x2b2		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75
#	0x6ef723	net/http.HandlerFunc.ServeHTTP+0x43									/usr/local/go/src/net/http/server.go:1947
#	0x1269d0b	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP+0x3ab		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:90
#	0x126bbd6	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1+0x4e6		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:126
#	0x6ef723	net/http.HandlerFunc.ServeHTTP+0x43									/usr/local/go/src/net/http/server.go:1947
#	0x126c243	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1+0xd3			/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47
#	0x6ef723	net/http.HandlerFunc.ServeHTTP+0x43									/usr/local/go/src/net/http/server.go:1947
#	0xfbd6a0	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1+0x180		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39
#	0x6ef723	net/http.HandlerFunc.ServeHTTP+0x43									/usr/local/go/src/net/http/server.go:1947
#	0x126c657	k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithPanicRecovery.func1+0x107		/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41
#	0x6ef723	net/http.HandlerFunc.ServeHTTP+0x43									/usr/local/go/src/net/http/server.go:1947
@MHBauer

This comment has been minimized.

Copy link
Member

MHBauer commented Aug 1, 2018

Lastly, if you restart on one side, the other side does clean up. At least if it's the kubelet.

I restarted the kubelet, and apiserver shows

goroutine profile: total 1859

Which is the ~600 waiting goroutines.

@MHBauer

This comment has been minimized.

Copy link
Member

MHBauer commented Aug 10, 2018

Based on some conversation with @Random-Liu I will look into the approach here for this issue. That issue (#64315) seems like a similar problem.

hh pushed a commit to ii/kubernetes that referenced this issue Aug 17, 2018

Merge pull request kubernetes#67288 from MHBauer/fast-close
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

immediately close the other side of the connection when proxying

**What this PR does / why we need it**:
close the other side of the connection by exiting once one side closes. This code is used in both apiserver and kubelet.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes kubernetes#57992

**Special notes for your reviewer**:
unit tests passed locally. something similar is done in client-go/tools/portforward. doesn't seem to break the tests.

**Release note**:
```release-note
NONE
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment