Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Severe memory leak in beta.470 and beyond crashes Traefik server #462

Closed
tooda02 opened this issue Jun 16, 2016 · 11 comments · Fixed by #464
Closed

Severe memory leak in beta.470 and beyond crashes Traefik server #462

tooda02 opened this issue Jun 16, 2016 · 11 comments · Fixed by #464
Labels
kind/bug/confirmed a confirmed bug (reproducible). status/5-frozen-due-to-age
Milestone
1.0

Comments

@tooda02
Copy link
Contributor

tooda02 commented Jun 16, 2016

Cisco recently attempted to deploy rc2 in production and was forced to back off due to a severe memory leak that brought down the entire VM running Traefik after about 14 hours or so. We've done some research and found that this was introduced in v1.0.0-beta.470: Merge pull request #305 from containous/fix-races. Here is a comparison between beta.470 and beta.453. (Beta453 does not exhibit the leak behavior).

The issue may be Marathon related. We are using the Marathon provider and currently have an issue that its configuration refreshes very frequently due to instability in some of the tasks it's running. We currently suspect PR #305 Fix races, but are still researching to narrow it down. I will update this issue with additional information as we find it.

This issue is a complete showstopper for us.
@emilevauge emilevauge added the bug label Jun 16, 2016
@emilevauge emilevauge added this to the 1.0 milestone Jun 16, 2016
@emilevauge
Copy link
Member

Hi @tooda02
Sorry for the inconvenience.
Could you provide your traefik.toml file?
Trying to reproduce using marathon backend.

@emilevauge emilevauge mentioned this issue Jun 16, 2016
Closed
@tooda02
Copy link
Contributor Author

tooda02 commented Jun 16, 2016

Here is traefik.toml

port = ":80"
graceTimeOut = 10
logLevel = "ERROR"
traefikLogsFile = "/tmp/log/traefik/traefik.log"
accessLogsFile = ""

[web]
address = ":8081"
CertFile = "/etc/pki/tls/certs/host.cert"
KeyFile = "/etc/pki/tls/private/host.key"

[marathon]
endpoint = "http://marathon.service.consul:18080"
domain = "prd.shipped-cisco.com"
watch = true
networkInterface = "eth0"
ExposedByDefault = true

I have confirmed that the leak was introduced by PR #305.

@emilevauge
Copy link
Member

OK thanks. It seems from my tests, that the leak is not due to marathon backend. Still investigating.

@npateriya npateriya mentioned this issue Jun 16, 2016
Merged
3 tasks
@emilevauge
Copy link
Member

@tooda02 if that's possible, it would be helpful if, during a leak:

  • kill -SIGQUIT ${PID_OF_TRAEFIK}. This will kill traefik and print on stdout the goroutines stack.
  • copy the goroutines stack and provide it here

Thanks for your help :)

@tooda02
Copy link
Contributor Author

tooda02 commented Jun 16, 2016

Here is SIGQUIT output from killing the published beta.470 binary after about 10 minutes of execution and already showing 50% more memory usage that a pre-beta.470 version that had been running for days.

Also, prior to this I tried running a binary built from source modified from beta.470 by reverting the marathon.go changes. It still showed the leak behavior, so you're likely right that it's not Marathon.

SIGQUIT: quit
PC=0x477f93 m=1

goroutine 0 [idle]:
runtime.futex(0x131b1d0, 0x0, 0xc820035eb0, 0x0, 0x0, 0x477dbc, 0x3c, 0x0, 0x428efb, 0x131b1d0, ...)
    /usr/local/go/src/runtime/sys_linux_amd64.s:303 +0x23
runtime.futexsleep(0x131b1d0, 0x0, 0xdf8475800)
    /usr/local/go/src/runtime/os1_linux.go:57 +0xf0
runtime.notetsleep_internal(0x131b1d0, 0xdf8475800, 0xc820000480)
    /usr/local/go/src/runtime/lock_futex.go:174 +0x12b
runtime.notetsleep(0x131b1d0, 0xdf8475800, 0x9345a5c6bb4f1)
    /usr/local/go/src/runtime/lock_futex.go:194 +0x6b
runtime.sysmon()
    /usr/local/go/src/runtime/proc.go:3477 +0x487
runtime.mstart1()
    /usr/local/go/src/runtime/proc.go:1095 +0xec
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1065 +0x72

goroutine 1 [chan receive, 8 minutes]:
main.(*Server).Start(0xc820069e40)
    /go/src/github.com/containous/traefik/server.go:97 +0x146
main.run()
    /go/src/github.com/containous/traefik/cmd.go:217 +0x655
main.glob.func1(0x1313800, 0xc8200a95a0, 0x0, 0x2)
    /go/src/github.com/containous/traefik/cmd.go:27 +0x14
github.com/containous/traefik/vendor/github.com/spf13/cobra.(*Command).execute(0x1313800, 0xc82000a1f0, 0x2, 0x2, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/spf13/cobra/command.go:565 +0x85a
github.com/containous/traefik/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1313800, 0x1313800, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/spf13/cobra/command.go:651 +0x55c
github.com/containous/traefik/vendor/github.com/spf13/cobra.(*Command).Execute(0x1313800, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/spf13/cobra/command.go:610 +0x2d
main.main()
    /go/src/github.com/containous/traefik/traefik.go:11 +0x3f

goroutine 5 [syscall, 8 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 6 [sleep]:
time.Sleep(0x3b9aca00)
    /usr/local/go/src/runtime/time.go:59 +0xf9
github.com/containous/traefik/vendor/github.com/thoas/stats.New.func1(0xc82004aa20)
    /go/src/github.com/containous/traefik/vendor/github.com/thoas/stats/stats.go:33 +0x2e
created by github.com/containous/traefik/vendor/github.com/thoas/stats.New
    /go/src/github.com/containous/traefik/vendor/github.com/thoas/stats/stats.go:35 +0x166

goroutine 7 [select, 8 minutes, locked to thread]:
runtime.gopark(0xea33d0, 0xc820022f28, 0xd278b8, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820022f28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820022f28)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 8 [IO wait]:
net.runtime_pollWait(0x7f4303e981b8, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82016c680, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82016c680, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82016c620, 0x0, 0x7f4303e982d8, 0xc822218240)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820024960, 0xc82221c150, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820024960, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
github.com/containous/traefik/vendor/github.com/mailgun/manners.(*GracefulListener).Accept(0xc820172100, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/mailgun/manners/listener.go:73 +0x6b
net/http.(*Server).Serve(0xc82004cc80, 0x7f4303e98278, 0xc820172100, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
github.com/containous/traefik/vendor/github.com/mailgun/manners.(*GracefulServer).Serve(0xc820015700, 0x7f4303e98278, 0xc820172100, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/mailgun/manners/server.go:293 +0x3b0
github.com/containous/traefik/vendor/github.com/mailgun/manners.(*GracefulServer).ListenAndServe(0xc820015700, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/mailgun/manners/server.go:144 +0x20c
main.(*Server).startServer(0xc820069e40, 0xc820015700, 0xa, 0x0, 0x0, 0xc8200a97c0, 0x1c, 0xc8200c2f85, 0x5, 0xc82010b4a0, ...)
    /go/src/github.com/containous/traefik/server.go:313 +0x3d3
created by main.(*Server).startHTTPServers
    /go/src/github.com/containous/traefik/server.go:127 +0x51d

goroutine 9 [select]:
main.(*Server).listenProviders(0xc820069e40, 0xc82016c540)
    /go/src/github.com/containous/traefik/server.go:135 +0xc82
main.(*Server).Start.func1(0xc82016c540)
    /go/src/github.com/containous/traefik/server.go:89 +0x2a
github.com/containous/traefik/safe.(*Pool).Go.func1()
    /go/src/github.com/containous/traefik/safe/routine.go:31 +0x41
github.com/containous/traefik/safe.GoWithRecover.func1(0xea1e78, 0xc820165410)
    /go/src/github.com/containous/traefik/safe/routine.go:63 +0x44
created by github.com/containous/traefik/safe.GoWithRecover
    /go/src/github.com/containous/traefik/safe/routine.go:64 +0x3f

goroutine 10 [select]:
main.(*Server).listenConfigurations(0xc820069e40, 0xc82016c5b0)
    /go/src/github.com/containous/traefik/server.go:174 +0xafc
main.(*Server).Start.func2(0xc82016c5b0)
    /go/src/github.com/containous/traefik/server.go:92 +0x2a
github.com/containous/traefik/safe.(*Pool).Go.func1()
    /go/src/github.com/containous/traefik/safe/routine.go:31 +0x41
github.com/containous/traefik/safe.GoWithRecover.func1(0xea1e78, 0xc820165440)
    /go/src/github.com/containous/traefik/safe/routine.go:63 +0x44
created by github.com/containous/traefik/safe.GoWithRecover
    /go/src/github.com/containous/traefik/safe/routine.go:64 +0x3f

goroutine 943 [IO wait]:
net.runtime_pollWait(0x7f4303e97df8, 0x72, 0xc821c42000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8219cb100, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8219cb100, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8219cb0a0, 0xc821c42000, 0x1000, 0x1000, 0x0, 0x7f4303e92168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc821a116c8, 0xc821c42000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f4303e9c998, 0xc821a116c8, 0xc821c38068, 0xc821c42000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc821c237c0, 0xc821c42000, 0x1000, 0x1000, 0xc820218d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc8219f7ce0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8219f7ce0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc821c38000)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 25 [select]:
github.com/containous/traefik/provider.(*Marathon).Provide.func1(0xc8201f1340)
    /go/src/github.com/containous/traefik/provider/marathon.go:69 +0x2e0
github.com/containous/traefik/safe.(*Pool).Go.func1()
    /go/src/github.com/containous/traefik/safe/routine.go:31 +0x41
github.com/containous/traefik/safe.GoWithRecover.func1(0xea1e78, 0xc8202006f0)
    /go/src/github.com/containous/traefik/safe/routine.go:63 +0x44
created by github.com/containous/traefik/safe.GoWithRecover
    /go/src/github.com/containous/traefik/safe/routine.go:64 +0x3f

goroutine 13 [chan receive, 8 minutes]:
main.(*Server).listenSignals(0xc820069e40)
    /go/src/github.com/containous/traefik/server.go:257 +0x59
created by main.(*Server).Start
    /go/src/github.com/containous/traefik/server.go:96 +0x11f

goroutine 15 [chan send, 8 minutes]:
github.com/containous/traefik/vendor/github.com/mailgun/manners.(*GracefulServer).Serve.func1(0xc820015700, 0xc820172140, 0xc820172100)
    /go/src/github.com/containous/traefik/vendor/github.com/mailgun/manners/server.go:234 +0x40
created by github.com/containous/traefik/vendor/github.com/mailgun/manners.(*GracefulServer).Serve
    /go/src/github.com/containous/traefik/vendor/github.com/mailgun/manners/server.go:239 +0x26a

goroutine 23 [IO wait]:
net.runtime_pollWait(0x7f4303e980f8, 0x72, 0xc820208000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8201f1250, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201f1250, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201f11f0, 0xc820208000, 0x1000, 0x1000, 0x0, 0x7f4303e92168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820024b28, 0xc820208000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f4303e9c998, 0xc820024b28, 0xc820170478, 0xc820208000, 0x1000, 0x1000, 0x1ad, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8201fd560, 0xc820208000, 0x1000, 0x1000, 0x1ad, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).Read(0xc8201f4540, 0xc820208000, 0x1000, 0x1000, 0xc820039908, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:197 +0x126
net/http.(*body).readLocked(0xc8201f3f00, 0xc820208000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:651 +0xa1
net/http.(*body).Read(0xc8201f3f00, 0xc820208000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:643 +0x110
net/http.(*bodyEOFSignal).Read(0xc8201f3f40, 0xc820208000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1609 +0x26a
github.com/containous/traefik/vendor/github.com/donovanhide/eventsource.(*normaliser).Read(0xc8201fd800, 0xc820208000, 0x1000, 0x1000, 0x1, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/donovanhide/eventsource/normalise.go:19 +0x87
bufio.(*Reader).fill(0xc8201f4a20)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8201f4a20, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
github.com/containous/traefik/vendor/github.com/donovanhide/eventsource.(*decoder).Decode(0xc820039df8, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/donovanhide/eventsource/decoder.go:37 +0x63
github.com/containous/traefik/vendor/github.com/donovanhide/eventsource.(*Stream).stream(0xc82004b500, 0x7f4303e9cb40, 0xc8201f3f40)
    /go/src/github.com/containous/traefik/vendor/github.com/donovanhide/eventsource/stream.go:108 +0x1cf
created by github.com/containous/traefik/vendor/github.com/donovanhide/eventsource.SubscribeWithRequest
    /go/src/github.com/containous/traefik/vendor/github.com/donovanhide/eventsource/stream.go:65 +0x1c4

goroutine 21 [select, 8 minutes]:
net/http.(*persistConn).readLoop(0xc820170410)
    /usr/local/go/src/net/http/transport.go:1178 +0xd52
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 24 [select]:
github.com/containous/traefik/vendor/github.com/gambol99/go-marathon.(*marathonClient).registerSSESubscription.func1(0xc82004b500, 0xc8200b8460)
    /go/src/github.com/containous/traefik/vendor/github.com/gambol99/go-marathon/subscription.go:176 +0x3ab
created by github.com/containous/traefik/vendor/github.com/gambol99/go-marathon.(*marathonClient).registerSSESubscription
    /go/src/github.com/containous/traefik/vendor/github.com/gambol99/go-marathon/subscription.go:187 +0x2cb

goroutine 22 [select, 8 minutes]:
net/http.(*persistConn).writeLoop(0xc820170410)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 20 [IO wait]:
net.runtime_pollWait(0x7f4303e98038, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8201f1020, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201f1020, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8201f0fc0, 0x0, 0x7f4303e982d8, 0xc821f8c140)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820024b00, 0x42a882, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc820024b00, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2427 +0x41
crypto/tls.(*listener).Accept(0xc8201fd000, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/tls.go:52 +0x60
net/http.(*Server).Serve(0xc8201ce280, 0x7f4303e9c7c8, 0xc8201fd000, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
net/http.(*Server).ListenAndServeTLS(0xc8201ce280, 0xc8200a98e0, 0x1c, 0xc8200a99a0, 0x1d, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2278 +0x39a
net/http.ListenAndServeTLS(0xc8200c31d0, 0x5, 0xc8200a98e0, 0x1c, 0xc8200a99a0, 0x1d, 0x7f4303e9c740, 0xc820153a90, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2228 +0xcd
main.(*WebProvider).Provide.func3(0xc8200155c0, 0xc820153a90)
    /go/src/github.com/containous/traefik/web.go:89 +0xc1
created by main.(*WebProvider).Provide
    /go/src/github.com/containous/traefik/web.go:99 +0x1588

goroutine 944 [select]:
net/http.(*persistConn).writeLoop(0xc821c38000)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 50 [IO wait]:
net.runtime_pollWait(0x7f4303e97d38, 0x72, 0xc8202d4000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82026ced0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82026ced0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82026ce70, 0xc8202d4000, 0x1000, 0x1000, 0x0, 0x7f4303e92168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8200252c8, 0xc8202d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f4303e9c998, 0xc8200252c8, 0xc820419178, 0xc8202d4000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc820568b80, 0xc8202d4000, 0x1000, 0x1000, 0xc82003dd1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820234840)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820234840, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820419110)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 41 [IO wait]:
net.runtime_pollWait(0x7f4303e97eb8, 0x72, 0xc8201d9000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8201f0140, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201f0140, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201f00e0, 0xc8201d9000, 0x1000, 0x1000, 0x0, 0x7f4303e92168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820024040, 0xc8201d9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f4303e9c998, 0xc820024040, 0xc820170208, 0xc8201d9000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc820378a20, 0xc8201d9000, 0x1000, 0x1000, 0xc820219d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc82037a420)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82037a420, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8201701a0)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 42 [select]:
net/http.(*persistConn).writeLoop(0xc8201701a0)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 51 [select]:
net/http.(*persistConn).writeLoop(0xc820419110)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 55 [IO wait]:
net.runtime_pollWait(0x7f4303e97c78, 0x72, 0xc8202d8000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82026d090, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82026d090, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82026d030, 0xc8202d8000, 0x1000, 0x1000, 0x0, 0x7f4303e92168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8200252e8, 0xc8202d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f4303e9c998, 0xc8200252e8, 0xc8204194b8, 0xc8202d8000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc820569620, 0xc8202d8000, 0x1000, 0x1000, 0xc820217d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820234d80)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820234d80, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820419450)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 56 [select]:
net/http.(*persistConn).writeLoop(0xc820419450)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 615 [IO wait, 4 minutes]:
net.runtime_pollWait(0x7f4303e978b8, 0x72, 0xc820d65000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8201f1fe0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201f1fe0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201f1f80, 0xc820d65000, 0x1000, 0x1000, 0x0, 0x7f4303e92168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8200240e8, 0xc820d65000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f4303e9c998, 0xc8200240e8, 0xc8208b33e8, 0xc820d65000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8215ce5c0, 0xc820d65000, 0x1000, 0x1000, 0xc8216fad1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820998420)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820998420, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8208b3380)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 616 [select, 4 minutes]:
net/http.(*persistConn).writeLoop(0xc8208b3380)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

rax    0xfffffffffffffffc
rbx    0xc820035eb0
rcx    0xffffffffffffffff
rdx    0x0
rdi    0x131b1d0
rsi    0x0
rbp    0x0
rsp    0xc820035e78
r8     0x0
r9     0x0
r10    0xc820035eb0
r11    0x246
r12    0x44a170
r13    0xea0220
r14    0x0
r15    0x8
rip    0x477f93
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

@tooda02
Copy link
Contributor Author

tooda02 commented Jun 16, 2016

Here is a second dump, this time after running about two hours and with the Traefik process taking approximately ten times the memory as that in the previous dump.

SIGQUIT: quit
PC=0x477f93 m=1

goroutine 0 [idle]:
runtime.futex(0x131b1d0, 0x0, 0xc820035eb0, 0x0, 0x0, 0x477dbc, 0x3c, 0x0, 0x428efb, 0x131b1d0, ...)
    /usr/local/go/src/runtime/sys_linux_amd64.s:303 +0x23
runtime.futexsleep(0x131b1d0, 0x0, 0xdf8475800)
    /usr/local/go/src/runtime/os1_linux.go:57 +0xf0
runtime.notetsleep_internal(0x131b1d0, 0xdf8475800, 0xc820000480)
    /usr/local/go/src/runtime/lock_futex.go:174 +0x12b
runtime.notetsleep(0x131b1d0, 0xdf8475800, 0x93acf064b5601)
    /usr/local/go/src/runtime/lock_futex.go:194 +0x6b
runtime.sysmon()
    /usr/local/go/src/runtime/proc.go:3477 +0x487
runtime.mstart1()
    /usr/local/go/src/runtime/proc.go:1095 +0xec
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1065 +0x72

goroutine 1 [chan receive, 63 minutes]:
main.(*Server).Start(0xc820069e40)
    /go/src/github.com/containous/traefik/server.go:97 +0x146
main.run()
    /go/src/github.com/containous/traefik/cmd.go:217 +0x655
main.glob.func1(0x1313800, 0xc8200a9580, 0x0, 0x2)
    /go/src/github.com/containous/traefik/cmd.go:27 +0x14
github.com/containous/traefik/vendor/github.com/spf13/cobra.(*Command).execute(0x1313800, 0xc82000a1f0, 0x2, 0x2, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/spf13/cobra/command.go:565 +0x85a
github.com/containous/traefik/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1313800, 0x1313800, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/spf13/cobra/command.go:651 +0x55c
github.com/containous/traefik/vendor/github.com/spf13/cobra.(*Command).Execute(0x1313800, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/spf13/cobra/command.go:610 +0x2d
main.main()
    /go/src/github.com/containous/traefik/traefik.go:11 +0x3f

goroutine 5 [syscall, 63 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 6 [sleep]:
time.Sleep(0x3b9aca00)
    /usr/local/go/src/runtime/time.go:59 +0xf9
github.com/containous/traefik/vendor/github.com/thoas/stats.New.func1(0xc82004a840)
    /go/src/github.com/containous/traefik/vendor/github.com/thoas/stats/stats.go:33 +0x2e
created by github.com/containous/traefik/vendor/github.com/thoas/stats.New
    /go/src/github.com/containous/traefik/vendor/github.com/thoas/stats/stats.go:35 +0x166

goroutine 7 [select, 63 minutes, locked to thread]:
runtime.gopark(0xea33d0, 0xc820022f28, 0xd278b8, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820022f28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820022f28)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 8 [IO wait]:
net.runtime_pollWait(0x7f90b25ef1b8, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82016c680, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82016c680, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82016c620, 0x0, 0x7f90b25ef2d8, 0xc82487eb20)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820024958, 0xc82a68c8a0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820024958, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
github.com/containous/traefik/vendor/github.com/mailgun/manners.(*GracefulListener).Accept(0xc820170140, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/mailgun/manners/listener.go:73 +0x6b
net/http.(*Server).Serve(0xc82004cc80, 0x7f90b25ef278, 0xc820170140, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
github.com/containous/traefik/vendor/github.com/mailgun/manners.(*GracefulServer).Serve(0xc8200156c0, 0x7f90b25ef278, 0xc820170140, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/mailgun/manners/server.go:293 +0x3b0
github.com/containous/traefik/vendor/github.com/mailgun/manners.(*GracefulServer).ListenAndServe(0xc8200156c0, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/mailgun/manners/server.go:144 +0x20c
main.(*Server).startServer(0xc820069e40, 0xc8200156c0, 0xa, 0x0, 0x0, 0xc8200a9800, 0x1c, 0xc8200c2f45, 0x5, 0xc8201094a0, ...)
    /go/src/github.com/containous/traefik/server.go:313 +0x3d3
created by main.(*Server).startHTTPServers
    /go/src/github.com/containous/traefik/server.go:127 +0x51d

goroutine 9 [select]:
main.(*Server).listenProviders(0xc820069e40, 0xc82016c540)
    /go/src/github.com/containous/traefik/server.go:135 +0xc82
main.(*Server).Start.func1(0xc82016c540)
    /go/src/github.com/containous/traefik/server.go:89 +0x2a
github.com/containous/traefik/safe.(*Pool).Go.func1()
    /go/src/github.com/containous/traefik/safe/routine.go:31 +0x41
github.com/containous/traefik/safe.GoWithRecover.func1(0xea1e78, 0xc820163410)
    /go/src/github.com/containous/traefik/safe/routine.go:63 +0x44
created by github.com/containous/traefik/safe.GoWithRecover
    /go/src/github.com/containous/traefik/safe/routine.go:64 +0x3f

goroutine 10 [select]:
main.(*Server).listenConfigurations(0xc820069e40, 0xc82016c5b0)
    /go/src/github.com/containous/traefik/server.go:174 +0xafc
main.(*Server).Start.func2(0xc82016c5b0)
    /go/src/github.com/containous/traefik/server.go:92 +0x2a
github.com/containous/traefik/safe.(*Pool).Go.func1()
    /go/src/github.com/containous/traefik/safe/routine.go:31 +0x41
github.com/containous/traefik/safe.GoWithRecover.func1(0xea1e78, 0xc820163440)
    /go/src/github.com/containous/traefik/safe/routine.go:63 +0x44
created by github.com/containous/traefik/safe.GoWithRecover
    /go/src/github.com/containous/traefik/safe/routine.go:64 +0x3f

goroutine 1326 [IO wait]:
net.runtime_pollWait(0x7f90b25ee678, 0x72, 0xc823f62000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820041100, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820041100, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8200410a0, 0xc823f62000, 0x1000, 0x1000, 0x0, 0x7f90b25e9168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820e136f0, 0xc823f62000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f90b25f3998, 0xc820e136f0, 0xc823ce34b8, 0xc823f62000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc82232fc40, 0xc823f62000, 0x1000, 0x1000, 0xc822322d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc823f4e780)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc823f4e780, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc823ce3450)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 25 [select]:
github.com/containous/traefik/provider.(*Marathon).Provide.func1(0xc8201f1340)
    /go/src/github.com/containous/traefik/provider/marathon.go:69 +0x2e0
github.com/containous/traefik/safe.(*Pool).Go.func1()
    /go/src/github.com/containous/traefik/safe/routine.go:31 +0x41
github.com/containous/traefik/safe.GoWithRecover.func1(0xea1e78, 0xc8202026c0)
    /go/src/github.com/containous/traefik/safe/routine.go:63 +0x44
created by github.com/containous/traefik/safe.GoWithRecover
    /go/src/github.com/containous/traefik/safe/routine.go:64 +0x3f

goroutine 13 [chan receive, 63 minutes]:
main.(*Server).listenSignals(0xc820069e40)
    /go/src/github.com/containous/traefik/server.go:257 +0x59
created by main.(*Server).Start
    /go/src/github.com/containous/traefik/server.go:96 +0x11f

goroutine 15 [chan send, 63 minutes]:
github.com/containous/traefik/vendor/github.com/mailgun/manners.(*GracefulServer).Serve.func1(0xc8200156c0, 0xc820170180, 0xc820170140)
    /go/src/github.com/containous/traefik/vendor/github.com/mailgun/manners/server.go:234 +0x40
created by github.com/containous/traefik/vendor/github.com/mailgun/manners.(*GracefulServer).Serve
    /go/src/github.com/containous/traefik/vendor/github.com/mailgun/manners/server.go:239 +0x26a

goroutine 23 [IO wait]:
net.runtime_pollWait(0x7f90b25eef78, 0x72, 0xc820208000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8201f1250, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201f1250, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201f11f0, 0xc820208000, 0x1000, 0x1000, 0x0, 0x7f90b25e9168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820024b00, 0xc820208000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f90b25f3998, 0xc820024b00, 0xc82016ad68, 0xc820208000, 0x1000, 0x1000, 0x1f7, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8201ff520, 0xc820208000, 0x1000, 0x1000, 0x1f7, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).Read(0xc8201f4360, 0xc820208000, 0x1000, 0x1000, 0xc820039908, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:197 +0x126
net/http.(*body).readLocked(0xc8201f3e00, 0xc820208000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:651 +0xa1
net/http.(*body).Read(0xc8201f3e00, 0xc820208000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:643 +0x110
net/http.(*bodyEOFSignal).Read(0xc8201f3e40, 0xc820208000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1609 +0x26a
github.com/containous/traefik/vendor/github.com/donovanhide/eventsource.(*normaliser).Read(0xc8201ff780, 0xc820208000, 0x1000, 0x1000, 0x1, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/donovanhide/eventsource/normalise.go:19 +0x87
bufio.(*Reader).fill(0xc8201f4840)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8201f4840, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
github.com/containous/traefik/vendor/github.com/donovanhide/eventsource.(*decoder).Decode(0xc820039df8, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/containous/traefik/vendor/github.com/donovanhide/eventsource/decoder.go:37 +0x63
github.com/containous/traefik/vendor/github.com/donovanhide/eventsource.(*Stream).stream(0xc82004b320, 0x7f90b25f3b40, 0xc8201f3e40)
    /go/src/github.com/containous/traefik/vendor/github.com/donovanhide/eventsource/stream.go:108 +0x1cf
created by github.com/containous/traefik/vendor/github.com/donovanhide/eventsource.SubscribeWithRequest
    /go/src/github.com/containous/traefik/vendor/github.com/donovanhide/eventsource/stream.go:65 +0x1c4

goroutine 21 [select, 63 minutes]:
net/http.(*persistConn).readLoop(0xc82016ad00)
    /usr/local/go/src/net/http/transport.go:1178 +0xd52
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 22 [select, 63 minutes]:
net/http.(*persistConn).writeLoop(0xc82016ad00)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 24 [select]:
github.com/containous/traefik/vendor/github.com/gambol99/go-marathon.(*marathonClient).registerSSESubscription.func1(0xc82004b320, 0xc8200b8460)
    /go/src/github.com/containous/traefik/vendor/github.com/gambol99/go-marathon/subscription.go:176 +0x3ab
created by github.com/containous/traefik/vendor/github.com/gambol99/go-marathon.(*marathonClient).registerSSESubscription
    /go/src/github.com/containous/traefik/vendor/github.com/gambol99/go-marathon/subscription.go:187 +0x2cb

goroutine 20 [IO wait]:
net.runtime_pollWait(0x7f90b25ef038, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8201f1020, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201f1020, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8201f0fc0, 0x0, 0x7f90b25ef2d8, 0xc828097440)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820024ad8, 0x42a882, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc820024ad8, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2427 +0x41
crypto/tls.(*listener).Accept(0xc8201fefc0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/tls.go:52 +0x60
net/http.(*Server).Serve(0xc8201ca280, 0x7f90b25f37c8, 0xc8201fefc0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
net/http.(*Server).ListenAndServeTLS(0xc8201ca280, 0xc8200a9920, 0x1c, 0xc8200a99e0, 0x1d, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2278 +0x39a
net/http.ListenAndServeTLS(0xc8200c3190, 0x5, 0xc8200a9920, 0x1c, 0xc8200a99e0, 0x1d, 0x7f90b25f3740, 0xc820151a90, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2228 +0xcd
main.(*WebProvider).Provide.func3(0xc820015580, 0xc820151a90)
    /go/src/github.com/containous/traefik/web.go:89 +0xc1
created by main.(*WebProvider).Provide
    /go/src/github.com/containous/traefik/web.go:99 +0x1588

goroutine 1324 [IO wait, 24 minutes]:
net.runtime_pollWait(0x7f90b25ee738, 0x72, 0xc823f60000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820040fb0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820040fb0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820040f50, 0xc823f60000, 0x1000, 0x1000, 0x0, 0x7f90b25e9168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820e136e8, 0xc823f60000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f90b25f3998, 0xc820e136e8, 0xc823ce3178, 0xc823f60000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc82232fbe0, 0xc823f60000, 0x1000, 0x1000, 0xc82020dd1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc823f4e5a0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc823f4e5a0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc823ce3110)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 6316 [select]:
net/http.(*persistConn).writeLoop(0xc8271fb860)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 44 [IO wait, 24 minutes]:
net.runtime_pollWait(0x7f90b25eeeb8, 0x72, 0xc8204f4000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820448ae0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820448ae0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820448a80, 0xc8204f4000, 0x1000, 0x1000, 0x0, 0x7f90b25e9168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820024158, 0xc8204f4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f90b25f3998, 0xc820024158, 0xc820071318, 0xc8204f4000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc82021ac20, 0xc8204f4000, 0x1000, 0x1000, 0xc82003bd1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820344c60)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820344c60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8200712b0)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 37 [IO wait, 5 minutes]:
net.runtime_pollWait(0x7f90b25eed38, 0x72, 0xc8202ca000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82050a5a0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82050a5a0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82050a540, 0xc8202ca000, 0x1000, 0x1000, 0x0, 0x7f90b25e9168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8200250c0, 0xc8202ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f90b25f3998, 0xc8200250c0, 0xc8204e93e8, 0xc8202ca000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8203ef340, 0xc8202ca000, 0x1000, 0x1000, 0xc82020ed1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc8205786c0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8205786c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8204e9380)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 38 [select, 5 minutes]:
net/http.(*persistConn).writeLoop(0xc8204e9380)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 45 [select, 24 minutes]:
net/http.(*persistConn).writeLoop(0xc8200712b0)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 56 [IO wait]:
net.runtime_pollWait(0x7f90b25eec78, 0x72, 0xc820126000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82019cd80, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82019cd80, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82019cd20, 0xc820126000, 0x1000, 0x1000, 0x0, 0x7f90b25e9168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820024e80, 0xc820126000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f90b25f3998, 0xc820024e80, 0xc82016a3a8, 0xc820126000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc82047bc00, 0xc820126000, 0x1000, 0x1000, 0xc82003dd1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820474480)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820474480, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc82016a340)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 57 [select]:
net/http.(*persistConn).writeLoop(0xc82016a340)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 1245 [IO wait, 24 minutes]:
net.runtime_pollWait(0x7f90b25eebb8, 0x72, 0xc821390000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82214e680, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82214e680, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82214e620, 0xc821390000, 0x1000, 0x1000, 0x0, 0x7f90b25e9168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820025f00, 0xc821390000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f90b25f3998, 0xc820025f00, 0xc822472a28, 0xc821390000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc823131be0, 0xc821390000, 0x1000, 0x1000, 0xc82003ad1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820e5b740)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820e5b740, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8224729c0)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 1246 [select, 24 minutes]:
net/http.(*persistConn).writeLoop(0xc8224729c0)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 10224 [select]:
net/http.(*persistConn).writeLoop(0xc8313bb450)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 1325 [select, 24 minutes]:
net/http.(*persistConn).writeLoop(0xc823ce3110)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 6315 [IO wait]:
net.runtime_pollWait(0x7f90b25ee7f8, 0x72, 0xc824f5e000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82af259c0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82af259c0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82af25960, 0xc824f5e000, 0x1000, 0x1000, 0x0, 0x7f90b25e9168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82e06c9f0, 0xc824f5e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f90b25f3998, 0xc82e06c9f0, 0xc8271fb8c8, 0xc824f5e000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8272d4dc0, 0xc824f5e000, 0x1000, 0x1000, 0xc822324d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc824cd8240)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc824cd8240, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8271fb860)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 1327 [select]:
net/http.(*persistConn).writeLoop(0xc823ce3450)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 10223 [IO wait]:
net.runtime_pollWait(0x7f90b25eea38, 0x72, 0xc829faf000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82a20a5a0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82a20a5a0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82a20a540, 0xc829faf000, 0x1000, 0x1000, 0x0, 0x7f90b25e9168, 0xc820010200)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82bff7998, 0xc829faf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f90b25f3998, 0xc82bff7998, 0xc8313bb4b8, 0xc829faf000, 0x1000, 0x1000, 0x41edf3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc836e084c0, 0xc829faf000, 0x1000, 0x1000, 0xc820210d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc831e7cba0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc831e7cba0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8313bb450)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

rax    0xfffffffffffffffc
rbx    0xc820035eb0
rcx    0xffffffffffffffff
rdx    0x0
rdi    0x131b1d0
rsi    0x0
rbp    0x0
rsp    0xc820035e78
r8     0x0
r9     0x0
r10    0xc820035eb0
r11    0x246
r12    0x44a170
r13    0xea0220
r14    0x0
r15    0x8
rip    0x477f93
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

@emilevauge
Copy link
Member

emilevauge commented Jun 16, 2016 •
edited

Thanks a lot @tooda02 :)
It does not seem to be linked to go routines, I may have found the issue.
Making a PR soon.

@emilevauge emilevauge mentioned this issue Jun 16, 2016
Merged
@tooda02
Copy link
Contributor Author

tooda02 commented Jun 16, 2016

I've tested PR Fix memory leak in listenProviders #464 on the system where I encountered the problem, and I no longer see a memory leak. Thanks very much for the quick and effective response.

@emilevauge
Copy link
Member

@emilevauge
Copy link
Member

For those who want to test the fix from PR #464, you can grab the docker image containous/traefik:pr-464.

@shouze
Copy link

shouze commented Jun 17, 2016

@emilevauge yes, tested & looks 👍

@ldez ldez added the kind/bug/confirmed a confirmed bug (reproducible). label Apr 29, 2017
@traefik traefik locked and limited conversation to collaborators Sep 1, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug/confirmed a confirmed bug (reproducible). status/5-frozen-due-to-age
Projects
None yet
Milestone
1.0
Development

Successfully merging a pull request may close this issue.

Fix memory leak in listenProviders traefik/traefik
5 participants
@shouze @ldez @emilevauge @tooda02 @traefiker