New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go routine leak #1233

Closed
miekg opened this Issue Nov 13, 2017 · 5 comments

Comments

Projects
None yet
1 participant
@miekg
Member

miekg commented Nov 13, 2017

See: https://snapshot.raintank.io/dashboard/snapshot/OQ0upeF7EFPoRW2Dfgl9XHzD0c7OSZXY

This is my local proxy.

config:

. {
	proxy . 8.8.8.8:53 {
		protocol https_google
	}
        cache
	prometheus
	log
	errors
}

Only diff. with my auth server is cache and proxy so either one is suspect.

@miekg miekg added the bug label Nov 13, 2017

@miekg

This comment has been minimized.

Member

miekg commented Nov 14, 2017

May be related to #1235

@miekg

This comment has been minimized.

Member

miekg commented Nov 14, 2017

Think it is the healthchecking

@miekg

This comment has been minimized.

Member

miekg commented Nov 16, 2017

dumping go-routines. This first one is increasing

goroutine profile: total 27
7 @ 0x3e6c8 0x3e790 0x14124 0x13ea8 0x224060 0x6fd2c
#	0x22405f	github.com/miekg/dns.zlexer+0x347	/home/miek/g/src/github.com/miekg/dns/scan.go:602

2 @ 0x3e6c8 0x39a5c 0x38edc 0xa420c 0xa4280 0xa4e78 0x135948 0x145e04 0x1e48a0 0x1e4d14 0x1e7cd0 0x302a58 0xf8530 0xf868c 0x30337c 0x6fd2c
#	0x38edb		internal/poll.runtime_pollWait+0x43	/home/miek/upstream/go/src/runtime/netpoll.go:173
#	0xa420b		internal/poll.(*pollDesc).wait+0x8b	/home/miek/upstream/go/src/internal/poll/fd_poll_runtime.go:85
#	0xa427f		internal/poll.(*pollDesc).waitRead+0x2b	/home/miek/upstream/go/src/internal/poll/fd_poll_runtime.go:90
#	0xa4e77		internal/poll.(*FD).Read+0x147		/home/miek/upstream/go/src/internal/poll/fd_unix.go:125
#	0x135947	net.(*netFD).Read+0x37			/home/miek/upstream/go/src/net/fd_unix.go:202
#	0x145e03	net.(*conn).Read+0x57			/home/miek/upstream/go/src/net/net.go:176
#	0x1e489f	crypto/tls.(*block).readFromUntil+0x77	/home/miek/upstream/go/src/crypto/tls/conn.go:488
#	0x1e4d13	crypto/tls.(*Conn).readRecord+0x9b	/home/miek/upstream/go/src/crypto/tls/conn.go:590
#	0x1e7ccf	crypto/tls.(*Conn).Read+0xcb		/home/miek/upstream/go/src/crypto/tls/conn.go:1134
#	0x302a57	net/http.(*persistConn).Read+0x173	/home/miek/upstream/go/src/net/http/transport.go:1391
#	0xf852f		bufio.(*Reader).fill+0x103		/home/miek/upstream/go/src/bufio/bufio.go:97
#	0xf868b		bufio.(*Reader).Peek+0x27		/home/miek/upstream/go/src/bufio/bufio.go:129
#	0x30337b	net/http.(*persistConn).readLoop+0x157	/home/miek/upstream/go/src/net/http/transport.go:1539

2 @ 0x3e6c8 0x4e558 0x304680 0x6fd2c
#	0x30467f	net/http.(*persistConn).writeLoop+0xeb	/home/miek/upstream/go/src/net/http/transport.go:1759

1 @ 0x1f708 0x53a98 0x17be38 0x6fd2c
#	0x53a97		os/signal.signal_recv+0x133	/home/miek/upstream/go/src/runtime/sigqueue.go:131
#	0x17be37	os/signal.loop+0x13		/home/miek/upstream/go/src/os/signal/signal_unix.go:22
@miekg

This comment has been minimized.

Member

miekg commented Nov 16, 2017

Hmmm, it goes down as well.

@miekg

This comment has been minimized.

Member

miekg commented Nov 16, 2017

Using more minimal Corefile:

 {
	proxy . 8.8.8.8:53 {
                except xxx xxxxx xxx x yy zzz # some internal names
		protocol https_google
	}
#        cache
	prometheus
#	log
#	errors
	pprof
}

miekg added a commit to miekg/dns that referenced this issue Nov 16, 2017

Fix go-routine leak
When and higher level (grammar or syntax) error was encountered when
parsing records from strings the routine would return, but the openend
channel would be left open and no other error would be triggered; this
means the goroutine starting z.lexer would be left hanging.

The fix is a bit of a hack; as we signal the upper error back down via
s.e and the trigger an "error" in the lexer, this at least ensure the
z.lexer function return and cleans up the goroutine.

Fixes #586
Fixes coredns/coredns#1233

miekg added a commit to miekg/dns that referenced this issue Nov 16, 2017

txt parser: fix goroutine leak
When a higher level (grammar or syntax) error was encountered the lower
level zlexer routine would be left open. This leaks a goroutine.

This PR fixes this by signalling this error, retrieving any remaining
items from the channel, so zlexer can return.

Fixes #586
Fixes coredns/coredns#1233

miekg added a commit to miekg/dns that referenced this issue Nov 17, 2017

txt parser: fix goroutine leak
When a higher level (grammar or syntax) error was encountered the lower
level zlexer routine would be left open and trying to send more tokens
on the channel c. This leaks a goroutine, per failed parse...

This PR fixes this by signalling this error - by canceling a context -
retrieving any remaining items from the channel, so zlexer can return.

It also adds a goroutine leak test that can be re-used in other tests,
the TestParseBadNAPTR test uses this leak detector.

The private key parsing code had the same bug and is also fixed in this
PR.

Fixes #586
Fixes coredns/coredns#1233

miekg added a commit to miekg/dns that referenced this issue Nov 17, 2017

txt parser: fix goroutine leak (#570)
* txt parser: fix goroutine leak

When a higher level (grammar or syntax) error was encountered the lower
level zlexer routine would be left open and trying to send more tokens
on the channel c. This leaks a goroutine, per failed parse...

This PR fixes this by signalling this error - by canceling a context -
retrieving any remaining items from the channel, so zlexer can return.

It also adds a goroutine leak test that can be re-used in other tests,
the TestParseBadNAPTR test uses this leak detector.

The private key parsing code had the same bug and is also fixed in this
PR.

Fixes #586
Fixes coredns/coredns#1233

* sem not needed anymore
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment