Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix panic while server shutdown #4644

Merged
merged 1 commit into from Mar 21, 2019
Merged

Fix panic while server shutdown #4644

merged 1 commit into from Mar 21, 2019

Conversation

juliens
Copy link
Member

@juliens juliens commented Mar 21, 2019

What does this PR do?

Fix a concurrent map write and read.

Motivation

Fix a panic during Traefik shutdown.

Detail 


fatal error: concurrent map iteration and map write

goroutine 94 [running]:
runtime.throw(0x1fd042e, 0x26)
	/usr/local/go/src/runtime/panic.go:617 +0x72 fp=0xc00017fde0 sp=0xc00017fdb0 pc=0x42dae2
runtime.mapiternext(0xc00017fee8)
	/usr/local/go/src/runtime/map.go:860 +0x597 fp=0xc00017fe68 sp=0xc00017fde0 pc=0x40f097
runtime.mapiterinit(0x1c1dfe0, 0xc0005e2c00, 0xc00017fee8)
	/usr/local/go/src/runtime/map.go:850 +0x1c9 fp=0xc00017fe88 sp=0xc00017fe68 pc=0x40ea09
github.com/containous/traefik/pkg/server.(*connectionTracker).Close(0xc0005e1700)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:315 +0x78 fp=0xc00017ff58 sp=0xc00017fe88 pc=0x19d69e8
github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown.func3(0xc0003a43c0, 0xc0005dab40, 0x235c900, 0xc000612060, 0x239f7c0, 0xc00061a370)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:167 +0x179 fp=0xc00017ffb0 sp=0xc00017ff58 pc=0x19d85b9
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00017ffb8 sp=0xc00017ffb0 pc=0x45ce91
created by github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:162 +0x223

goroutine 1 [chan receive]:
github.com/containous/traefik/pkg/server.(*Server).Wait(...)
	/go/src/github.com/containous/traefik/pkg/server/server.go:151
main.runCmd(0xc00053ea80, 0xc0000562a0, 0x57, 0x0, 0x0)
	/go/src/github.com/containous/traefik/cmd/traefik/traefik.go:304 +0xdde
main.main.func1(0x0, 0xc00053ea80)
	/go/src/github.com/containous/traefik/cmd/traefik/traefik.go:102 +0x3c
github.com/containous/traefik/vendor/github.com/containous/staert.(*Staert).Run(...)
	/go/src/github.com/containous/traefik/vendor/github.com/containous/staert/staert.go:79
main.main()
	/go/src/github.com/containous/traefik/cmd/traefik/traefik.go:184 +0x15f6

goroutine 7 [chan receive]:
github.com/containous/traefik/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x381f7e0)
	/go/src/github.com/containous/traefik/vendor/github.com/golang/glog/glog.go:879 +0x8b
created by github.com/containous/traefik/vendor/github.com/golang/glog.init.0
	/go/src/github.com/containous/traefik/vendor/github.com/golang/glog/glog.go:410 +0x272

goroutine 9 [syscall]:
os/signal.signal_recv(0x233b800)
	/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 11 [runnable]:
sync.runtime_SemacquireMutex(0xc0000ae170, 0x1)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc0000ae16c)
	/usr/local/go/src/sync/mutex.go:134 +0x109
github.com/containous/traefik/vendor/github.com/sirupsen/logrus.(*MutexWrap).Lock(...)
	/go/src/github.com/containous/traefik/vendor/github.com/sirupsen/logrus/logger.go:44
github.com/containous/traefik/vendor/github.com/sirupsen/logrus.(*Entry).fireHooks(0xc0003b6410)
	/go/src/github.com/containous/traefik/vendor/github.com/sirupsen/logrus/entry.go:131 +0x124
github.com/containous/traefik/vendor/github.com/sirupsen/logrus.Entry.log(0xc0000ae140, 0xc0006c8270, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000000002, ...)
	/go/src/github.com/containous/traefik/vendor/github.com/sirupsen/logrus/entry.go:111 +0xf9
github.com/containous/traefik/vendor/github.com/sirupsen/logrus.(*Entry).Error(0xc0003b6320, 0xc0005b4620, 0x1, 0x1)
	/go/src/github.com/containous/traefik/vendor/github.com/sirupsen/logrus/entry.go:181 +0x111
github.com/containous/traefik/vendor/github.com/sirupsen/logrus.(*Logger).Error(0xc0000ae140, 0xc0005b4620, 0x1, 0x1)
	/go/src/github.com/containous/traefik/vendor/github.com/sirupsen/logrus/logger.go:228 +0x74
github.com/containous/traefik/pkg/log.Error(...)
	/go/src/github.com/containous/traefik/pkg/log/deprecated.go:50
github.com/containous/traefik/pkg/server.(*TCPEntryPoint).startTCP(0xc0005dab40, 0x235c940, 0xc0005e7a40)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:103 +0x1bc
created by github.com/containous/traefik/pkg/server.(*Server).startTCPServers
	/go/src/github.com/containous/traefik/pkg/server/server.go:216 +0x276

goroutine 20 [sleep]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:307
time.Sleep(0x8bb2c97000)
	/usr/local/go/src/runtime/time.go:105 +0x159
main.checkNewVersion.func1()
	/go/src/github.com/containous/traefik/cmd/traefik/traefik.go:371 +0x39
github.com/containous/traefik/pkg/safe.GoWithRecover.func1(0x2056898, 0xc000490c90)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:142 +0x4d
created by github.com/containous/traefik/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:136 +0x49

goroutine 21 [sleep]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:307
time.Sleep(0x8bb2c97000)
	/usr/local/go/src/runtime/time.go:105 +0x159
main.collect.func1()
	/go/src/github.com/containous/traefik/cmd/traefik/traefik.go:406 +0x46
github.com/containous/traefik/pkg/safe.GoWithRecover.func1(0x2056898, 0xc0005e16e0)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:142 +0x4d
created by github.com/containous/traefik/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:136 +0x49

goroutine 22 [chan receive]:
github.com/containous/traefik/pkg/server.(*httpForwarder).Accept(0xc0005e1740, 0xc00086ce40, 0x18, 0xc000372f00, 0x6fb464)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:41 +0x41
net/http.(*Server).Serve(0xc000571ba0, 0x2341c00, 0xc0005e1740, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2859 +0x22d
github.com/containous/traefik/pkg/h2c.Server.Serve(0xc000571ba0, 0x2341c00, 0xc0005e1740, 0x0, 0x0)
	/go/src/github.com/containous/traefik/pkg/h2c/h2c.go:80 +0xb5
github.com/containous/traefik/pkg/server.createHTTPServer.func1(0x23419c0, 0xc00048ea48, 0xc0005e1740)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:361 +0x4b
created by github.com/containous/traefik/pkg/server.createHTTPServer
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:360 +0x29c

goroutine 23 [chan receive]:
github.com/containous/traefik/pkg/server.(*httpForwarder).Accept(0xc0005e1780, 0x1e2bf00, 0xc0006c8210, 0x1c44800, 0x3803c20)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:41 +0x41
net/http.(*Server).Serve(0xc000571c70, 0x2341c00, 0xc0005e1780, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2859 +0x22d
github.com/containous/traefik/pkg/server.createHTTPServer.func1(0x234f080, 0xc000571c70, 0xc0005e1780)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:361 +0x4b
created by github.com/containous/traefik/pkg/server.createHTTPServer
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:360 +0x29c

goroutine 24 [chan receive]:
github.com/containous/traefik/pkg/server.(*httpForwarder).Accept(0xc0005e1820, 0xc000868e40, 0x18, 0xc000373200, 0x6fb464)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:41 +0x41
net/http.(*Server).Serve(0xc000571e10, 0x2341c00, 0xc0005e1820, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2859 +0x22d
github.com/containous/traefik/pkg/h2c.Server.Serve(0xc000571e10, 0x2341c00, 0xc0005e1820, 0x0, 0x0)
	/go/src/github.com/containous/traefik/pkg/h2c/h2c.go:80 +0xb5
github.com/containous/traefik/pkg/server.createHTTPServer.func1(0x23419c0, 0xc00048ea68, 0xc0005e1820)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:361 +0x4b
created by github.com/containous/traefik/pkg/server.createHTTPServer
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:360 +0x29c

goroutine 25 [chan receive]:
github.com/containous/traefik/pkg/server.(*httpForwarder).Accept(0xc0005e1860, 0x1e2bf00, 0xc0006c8120, 0x1c44800, 0x3803c20)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:41 +0x41
net/http.(*Server).Serve(0xc000571ee0, 0x2341c00, 0xc0005e1860, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2859 +0x22d
github.com/containous/traefik/pkg/server.createHTTPServer.func1(0x234f080, 0xc000571ee0, 0xc0005e1860)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:361 +0x4b
created by github.com/containous/traefik/pkg/server.createHTTPServer
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:360 +0x29c

goroutine 85 [semacquire]:
sync.runtime_Semacquire(0xc0005600d8)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc0005600d0)
	/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown(0xc0005dabe0, 0x235c900, 0xc00008e780)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:173 +0x1a0
github.com/containous/traefik/pkg/server.(*Server).Stop.func1(0xc00004e3d0, 0xc0006202b7, 0x3, 0xc0005dabe0)
	/go/src/github.com/containous/traefik/pkg/server/server.go:165 +0x110
created by github.com/containous/traefik/pkg/server.(*Server).Stop
	/go/src/github.com/containous/traefik/pkg/server/server.go:161 +0x1a0

goroutine 84 [semacquire]:
sync.runtime_Semacquire(0xc0003a43c8)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc0003a43c0)
	/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown(0xc0005dab40, 0x235c900, 0xc000612060)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:173 +0x1a0
github.com/containous/traefik/pkg/server.(*Server).Stop.func1(0xc00004e3d0, 0xc00062032e, 0x7, 0xc0005dab40)
	/go/src/github.com/containous/traefik/pkg/server/server.go:165 +0x110
created by github.com/containous/traefik/pkg/server.(*Server).Stop
	/go/src/github.com/containous/traefik/pkg/server/server.go:161 +0x1a0

goroutine 29 [semacquire]:
sync.runtime_Semacquire(0xc00004e3d8)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc00004e3d0)
	/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/containous/traefik/pkg/server.(*Server).Stop(0xc000644000)
	/go/src/github.com/containous/traefik/pkg/server/server.go:170 +0x1ca
github.com/containous/traefik/pkg/server.(*Server).Start.func1(0xc000644000, 0x235c880, 0xc000619d00)
	/go/src/github.com/containous/traefik/pkg/server/server.go:133 +0x170
created by github.com/containous/traefik/pkg/server.(*Server).Start
	/go/src/github.com/containous/traefik/pkg/server/server.go:127 +0x57

goroutine 78 [runnable]:
net/http.(*http2serverInternalState).startGracefulShutdown-fm()
	/usr/local/go/src/net/http/h2_bundle.go:3747
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1
created by net/http.(*Server).Shutdown
	/usr/local/go/src/net/http/server.go:2642 +0xd3

goroutine 13 [select]:
github.com/containous/traefik/pkg/server.(*Server).listenProviders(0xc000644000, 0xc0004d41c0)
	/go/src/github.com/containous/traefik/pkg/server/server.go:222 +0xc5
github.com/containous/traefik/pkg/server.(*Server).Start.func2(0xc0004d41c0)
	/go/src/github.com/containous/traefik/pkg/server/server.go:138 +0x34
github.com/containous/traefik/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:78 +0x37
github.com/containous/traefik/pkg/safe.GoWithRecover.func1(0x2056898, 0xc0005e7b60)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:142 +0x4d
created by github.com/containous/traefik/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:136 +0x49

goroutine 14 [select]:
github.com/containous/traefik/pkg/server.(*Server).listenConfigurations(0xc000644000, 0xc0004d4230)
	/go/src/github.com/containous/traefik/pkg/server/server_configuration.go:214 +0xe1
github.com/containous/traefik/pkg/server.(*Server).Start.func3(0xc0004d4230)
	/go/src/github.com/containous/traefik/pkg/server/server.go:141 +0x34
github.com/containous/traefik/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:78 +0x37
github.com/containous/traefik/pkg/safe.GoWithRecover.func1(0x2056898, 0xc0005e7b90)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:142 +0x4d
created by github.com/containous/traefik/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:136 +0x49

goroutine 67 [select]:
github.com/containous/traefik/pkg/server.(*Server).throttleProviderConfigReload(0xc000644000, 0x0, 0xc000692ba0, 0xc0004cc7e0, 0xc0004d4380)
	/go/src/github.com/containous/traefik/pkg/server/server_configuration.go:249 +0x1e4
github.com/containous/traefik/pkg/server.(*Server).preLoadConfiguration.func1(0xc0004d4380)
	/go/src/github.com/containous/traefik/pkg/server/server_configuration.go:201 +0x52
github.com/containous/traefik/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:78 +0x37
github.com/containous/traefik/pkg/safe.GoWithRecover.func1(0x2056898, 0xc0006c9a70)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:142 +0x4d
created by github.com/containous/traefik/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:136 +0x49

goroutine 16 [select]:
github.com/containous/traefik/pkg/server.(*Server).listenSignals(0xc000644000, 0xc0004d42a0)
	/go/src/github.com/containous/traefik/pkg/server/server_signals.go:18 +0xce
github.com/containous/traefik/pkg/server.(*Server).Start.func4(0xc0004d42a0)
	/go/src/github.com/containous/traefik/pkg/server/server.go:145 +0x34
github.com/containous/traefik/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:78 +0x37
github.com/containous/traefik/pkg/safe.GoWithRecover.func1(0x2056898, 0xc0005e7c80)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:142 +0x4d
created by github.com/containous/traefik/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:136 +0x49

goroutine 38 [runnable]:
internal/poll.runtime_Semacquire(0xc0000f80a8)
	/usr/local/go/src/runtime/sema.go:61 +0x39
internal/poll.(*FD).Close(0xc0000f8080, 0xc0000f8080, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:109 +0x90
net.(*netFD).Close(0xc0000f8080, 0xc000026d70, 0xc0005e17e8)
	/usr/local/go/src/net/fd_unix.go:184 +0x4f
net.(*conn).Close(0xc00048e000, 0x0, 0x0)
	/usr/local/go/src/net/net.go:201 +0x4b
github.com/containous/traefik/pkg/server.(*trackedConnection).Close(0xc000686020, 0xc000026e20, 0xc000026de8)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:388 +0x57
net/http.(*Server).closeIdleConns(0xc000571e10, 0x205a700)
	/usr/local/go/src/net/http/server.go:2691 +0x14c
net/http.(*Server).Shutdown(0xc000571e10, 0x235c900, 0xc00008e780, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2649 +0x151
github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown.func1(0xc0005600d0, 0xc0005dabe0, 0x235c900, 0xc00008e780, 0x239f7c0, 0xc0000ae640)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:132 +0x79
created by github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:130 +0x31d

goroutine 65 [syscall]:
syscall.Syscall6(0xe8, 0x8, 0xc0006ffd6c, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0xb, 0x7ed00000007, 0x1)
	/usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
github.com/containous/traefik/vendor/golang.org/x/sys/unix.EpollWait(0x8, 0xc0006ffd6c, 0x7, 0x7, 0xffffffffffffffff, 0xc0006ffd68, 0x0, 0xc000880900)
	/go/src/github.com/containous/traefik/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1544 +0x72
github.com/containous/traefik/vendor/gopkg.in/fsnotify%!e(MISSING)v1.(*fdPoller).wait(0xc000518b60, 0x2059100, 0xc000518b60, 0x2)
	/go/src/github.com/containous/traefik/vendor/gopkg.in/fsnotify.v1/inotify_poller.go:86 +0x91
github.com/containous/traefik/vendor/gopkg.in/fsnotify%!e(MISSING)v1.(*Watcher).readEvents(0xc0003b70e0)
	/go/src/github.com/containous/traefik/vendor/gopkg.in/fsnotify.v1/inotify.go:192 +0x18a
created by github.com/containous/traefik/vendor/gopkg.in/fsnotify%!e(MISSING)v1.NewWatcher
	/go/src/github.com/containous/traefik/vendor/gopkg.in/fsnotify.v1/inotify.go:61 +0x1e2

goroutine 66 [select]:
github.com/containous/traefik/pkg/provider/file.(*Provider).addWatcher.func1(0xc0004d4310)
	/go/src/github.com/containous/traefik/pkg/provider/file/file.go:103 +0x16a
github.com/containous/traefik/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:78 +0x37
github.com/containous/traefik/pkg/safe.GoWithRecover.func1(0x2056898, 0xc0006c8a80)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:142 +0x4d
created by github.com/containous/traefik/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:136 +0x49

goroutine 31 [runnable]:
net/http.(*connReader).lock(0xc00064e450)
	/usr/local/go/src/net/http/server.go:653 +0xb6
net/http.(*connReader).Read(0xc00064e450, 0xc00046c000, 0x1000, 0x1000, 0x300000002, 0xc00088a480, 0xc000863a10)
	/usr/local/go/src/net/http/server.go:789 +0x133
bufio.(*Reader).fill(0xc000692060)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadSlice(0xc000692060, 0xc00005c50a, 0xc00005c500, 0xc000863b00, 0x40bd39, 0xc00046e600, 0x100)
	/usr/local/go/src/bufio/bufio.go:356 +0x3d
bufio.(*Reader).ReadLine(0xc000692060, 0xc000863b08, 0xc000070700, 0x7ff2a3b5a6d0, 0x0, 0x0, 0x1ec40a0)
	/usr/local/go/src/bufio/bufio.go:385 +0x34
net/textproto.(*Reader).readLineSlice(0xc00064e570, 0xc00046e600, 0x42d101, 0xc000863b98, 0x19d8dcb, 0xc00048e000)
	/usr/local/go/src/net/textproto/reader.go:55 +0x6f
net/textproto.(*Reader).ReadLine(...)
	/usr/local/go/src/net/textproto/reader.go:36
net/http.readRequest(0xc000692060, 0x0, 0xc00046e600, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:968 +0x8d
net/http.(*conn).readRequest(0xc00049a0a0, 0x235c880, 0xc0004940c0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:967 +0x163
net/http.(*conn).serve(0xc00049a0a0, 0x235c880, 0xc0004940c0)
	/usr/local/go/src/net/http/server.go:1819 +0x6ad
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 68 [select]:
github.com/containous/traefik/vendor/github.com/eapache/channels.(*RingChannel).ringBuffer(0xc0006c9ad0)
	/go/src/github.com/containous/traefik/vendor/github.com/eapache/channels/ring_channel.go:87 +0x205
created by github.com/containous/traefik/vendor/github.com/eapache/channels.NewRingChannel
	/go/src/github.com/containous/traefik/vendor/github.com/eapache/channels/ring_channel.go:32 +0x1b4

goroutine 69 [select]:
github.com/containous/traefik/pkg/server.(*Server).throttleProviderConfigReload.func1(0xc0004d43f0)
	/go/src/github.com/containous/traefik/pkg/server/server_configuration.go:236 +0xda
github.com/containous/traefik/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:78 +0x37
github.com/containous/traefik/pkg/safe.GoWithRecover.func1(0x2056898, 0xc0006c9b00)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:142 +0x4d
created by github.com/containous/traefik/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:136 +0x49

goroutine 70 [select]:
github.com/containous/traefik/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc0005b5fd0, 0xc000385f40)
	/go/src/github.com/containous/traefik/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0xd5
created by github.com/containous/traefik/vendor/github.com/patrickmn/go-cache.runJanitor
	/go/src/github.com/containous/traefik/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xa9

goroutine 76 [IO wait]:
internal/poll.runtime_pollWait(0x7ff2a18edbc8, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000498f98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000498f80, 0xc0008f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000498f80, 0xc0008f9000, 0x1000, 0x1000, 0x1, 0x42f8fc, 0xc00017eb88)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000f3a8, 0xc0008f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
net/http.(*persistConn).Read(0xc00062c6c0, 0xc0008f9000, 0x1000, 0x1000, 0xc00017ec88, 0x404de5, 0xc0004cd200)
	/usr/local/go/src/net/http/transport.go:1524 +0x7b
bufio.(*Reader).fill(0xc00088f380)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc00088f380, 0x1, 0x0, 0x0, 0x1, 0xc000056400, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc00062c6c0)
	/usr/local/go/src/net/http/transport.go:1677 +0x1a3
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1357 +0xae8

goroutine 77 [select]:
net/http.(*persistConn).writeLoop(0xc00062c6c0)
	/usr/local/go/src/net/http/transport.go:1958 +0x113
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1358 +0xb0d

goroutine 87 [IO wait]:
internal/poll.runtime_pollWait(0x7ff2a18edd68, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000470598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000470580, 0xc00094f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000470580, 0xc00094f000, 0x1000, 0x1000, 0x417c1e, 0x7ff2a1994a00, 0x20300000000000)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00038e030, 0xc00094f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x69
github.com/containous/traefik/pkg/tcp.(*Conn).Read(0xc00064f290, 0xc00094f000, 0x1000, 0x1000, 0x76, 0xc00085f9f0, 0x42cedd)
	/go/src/github.com/containous/traefik/pkg/tcp/router.go:155 +0x122
net/http.(*connReader).Read(0xc00064f320, 0xc00094f000, 0x1000, 0x1000, 0xc00005a000, 0xc0008f11a0, 0x22c734f)
	/usr/local/go/src/net/http/server.go:787 +0x107
bufio.(*Reader).fill(0xc000692540)
	/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadSlice(0xc000692540, 0xa, 0x13, 0xc00085fb00, 0x40bf32, 0xc0003b3300, 0x100)
	/usr/local/go/src/bufio/bufio.go:356 +0x3d
bufio.(*Reader).ReadLine(0xc000692540, 0xc00085fb08, 0xc000103880, 0x7ff2a3b5a008, 0xc000914300, 0x40c5c8, 0x30)
	/usr/local/go/src/bufio/bufio.go:385 +0x34
net/textproto.(*Reader).readLineSlice(0xc0008f11a0, 0xc0003b3300, 0x22c768a, 0xc00085fb98, 0x19d8dcb, 0xc00038e030)
	/usr/local/go/src/net/textproto/reader.go:55 +0x6f
net/textproto.(*Reader).ReadLine(...)
	/usr/local/go/src/net/textproto/reader.go:36
net/http.readRequest(0xc000692540, 0x0, 0xc0003b3300, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:968 +0x8d
net/http.(*conn).readRequest(0xc00049a280, 0x235c880, 0xc000494540, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:967 +0x163
net/http.(*conn).serve(0xc00049a280, 0x235c880, 0xc000494540)
	/usr/local/go/src/net/http/server.go:1819 +0x6ad
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 90 [semacquire]:
sync.runtime_SemacquireMutex(0xc000571c34, 0xc000082100)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc000571c30)
	/usr/local/go/src/sync/mutex.go:134 +0x109
net/http.(*Server).trackConn(0xc000571ba0, 0xc00049a460, 0x2310100)
	/usr/local/go/src/net/http/server.go:2955 +0x3d
net/http.(*conn).setState(0xc00049a460, 0x237ac40, 0xc00064f980, 0x4)
	/usr/local/go/src/net/http/server.go:1714 +0xfb
net/http.(*conn).serve.func1(0xc00049a460)
	/usr/local/go/src/net/http/server.go:1774 +0xeb
net/http.(*conn).serve(0xc00049a460, 0x235c880, 0xc000494780)
	/usr/local/go/src/net/http/server.go:1839 +0xdcd
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 40 [runnable]:
sync.runtime_SemacquireMutex(0xc0000ae170, 0x0)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc0000ae16c)
	/usr/local/go/src/sync/mutex.go:134 +0x109
github.com/containous/traefik/vendor/github.com/sirupsen/logrus.(*MutexWrap).Lock(...)
	/go/src/github.com/containous/traefik/vendor/github.com/sirupsen/logrus/logger.go:44
github.com/containous/traefik/vendor/github.com/sirupsen/logrus.(*Entry).fireHooks(0xc0003b6190)
	/go/src/github.com/containous/traefik/vendor/github.com/sirupsen/logrus/entry.go:131 +0x124
github.com/containous/traefik/vendor/github.com/sirupsen/logrus.Entry.log(0xc0000ae140, 0xc00064f1a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7ff200000005, ...)
	/go/src/github.com/containous/traefik/vendor/github.com/sirupsen/logrus/entry.go:111 +0xf9
github.com/containous/traefik/vendor/github.com/sirupsen/logrus.(*Entry).Debug(0xc0000ae640, 0xc0009c7f38, 0x1, 0x1)
	/go/src/github.com/containous/traefik/vendor/github.com/sirupsen/logrus/entry.go:155 +0x111
github.com/containous/traefik/vendor/github.com/sirupsen/logrus.(*Entry).Debugf(0xc0000ae640, 0x1fd3a93, 0x28, 0xc0005b41c0, 0x1, 0x1)
	/go/src/github.com/containous/traefik/vendor/github.com/sirupsen/logrus/entry.go:203 +0xd6
github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown.func3(0xc0005600d0, 0xc0005dabe0, 0x235c900, 0xc00008e780, 0x239f7c0, 0xc0000ae640)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:166 +0x167
created by github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:162 +0x223

goroutine 43 [runnable]:
net/http.(*http2serverInternalState).startGracefulShutdown-fm()
	/usr/local/go/src/net/http/h2_bundle.go:3747
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1
created by net/http.(*Server).Shutdown
	/usr/local/go/src/net/http/server.go:2642 +0xd3

goroutine 92 [runnable]:
internal/poll.runtime_Semacquire(0xc000470aa8)
	/usr/local/go/src/runtime/sema.go:61 +0x39
internal/poll.(*FD).Close(0xc000470a80, 0xc000470a80, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:109 +0x90
net.(*netFD).Close(0xc000470a80, 0xc0009c9d70, 0xc0005e1708)
	/usr/local/go/src/net/fd_unix.go:184 +0x4f
net.(*conn).Close(0xc00038e040, 0x0, 0x0)
	/usr/local/go/src/net/net.go:201 +0x4b
github.com/containous/traefik/pkg/server.(*trackedConnection).Close(0xc00061e500, 0xc0009c9e20, 0xc0009c9de8)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:388 +0x57
net/http.(*Server).closeIdleConns(0xc000571ba0, 0x205a700)
	/usr/local/go/src/net/http/server.go:2691 +0x14c
net/http.(*Server).Shutdown(0xc000571ba0, 0x235c900, 0xc000612060, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2649 +0x151
github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown.func1(0xc0003a43c0, 0xc0005dab40, 0x235c900, 0xc000612060, 0x239f7c0, 0xc00061a370)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:132 +0x79
created by github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:130 +0x31d

goroutine 93 [runnable]:
github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown.func2(0xc0003a43c0, 0xc0005dab40, 0x235c900, 0xc000612060, 0x239f7c0, 0xc00061a370)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:146
created by github.com/containous/traefik/pkg/server.(*TCPEntryPoint).Shutdown
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:146 +0x29c

@traefiker traefiker added this to the 2.0 milestone Mar 21, 2019
ldez
ldez approved these changes Mar 21, 2019
View reviewed changes
Copy link
Member

@ldez ldez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

jbdoumenjou
jbdoumenjou approved these changes Mar 21, 2019
View reviewed changes
Copy link
Member

@jbdoumenjou jbdoumenjou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

mmatur
mmatur approved these changes Mar 21, 2019
View reviewed changes
Copy link
Member

@mmatur mmatur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@traefiker traefiker merged commit f8f7edd into traefik:v2.0 Mar 21, 2019
@juliens juliens deleted the fix-race branch January 7, 2020 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jbdoumenjou jbdoumenjou jbdoumenjou approved these changes

@mmatur mmatur mmatur approved these changes

@ldez ldez ldez approved these changes

Assignees
No one assigned
Labels
area/server kind/bug/fix a bug fix size/S
Projects
None yet
Milestone
2.0
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@juliens @jbdoumenjou @mmatur @ldez @traefiker