Docker container with fluentd logging driver never stops if it can't connect to fluentd

[abicky]

Description

A docker container with fluentd logging driver never stops if it can't connect to the fluentd.
In such a case, docker kill can't stop the container, either.

Reproduce

1. Start a fluentd container

   docker run -d --rm --name fluentd -p 24224:24224 fluent/fluentd:v1.15.3-1.0
   

2. Start a container with fluentd logging driver

   docker run --rm --name hello --log-driver=fluentd --log-opt fluentd-address=$(hostname -i):24224 ubuntu \
     sh -c 'while :; do echo "$(date +%FT%T): hello"; sleep 1; done'
   

3. Stop the fluent container

   docker stop fluentd
   

4. Stop the other container

   docker stop hello
   

Even if we execute docker kill hello, the container "hello" never stops.

Expected behavior

The container "hello" should stop within the timeout.

docker version

Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:58 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:49 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.12.2)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-1019-aws
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 941.2MiB
 Name: ip-172-31-12-189
 ID: PIFL:VJRE:S4MN:3WUZ:WZ3B:HDS2:PS3N:V5HX:6JUV:36FV:63JL:7B4Y
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Here is the log after I executed docker stop hello:

Nov 22 19:12:29 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:29.695802580Z" level=debug msg="Calling HEAD /_ping"
Nov 22 19:12:29 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:29.696764997Z" level=debug msg="Calling POST /v1.41/containers/hello/stop"
Nov 22 19:12:29 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:29.696848361Z" level=debug msg="Sending kill signal 15 to container 6978d7c0917634b5493c124fad2d510397685abccea54fcdbc5a49cf86831618"
Nov 22 19:12:39 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:39.720093108Z" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=6978d7c0917634b5493c124fad2d510397685abccea54fcdbc5a49cf86831618
Nov 22 19:12:39 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:39.720141148Z" level=debug msg="Sending kill signal 9 to container 6978d7c0917634b5493c124fad2d510397685abccea54fcdbc5a49cf86831618"
Nov 22 19:12:39 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:39.738406188Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/exit
Nov 22 19:12:39 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:39.751965365Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/delete
Nov 22 19:12:39 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:39.751996631Z" level=info msg="ignoring event" container=6978d7c0917634b5493c124fad2d510397685abccea54fcdbc5a49cf86831618 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 22 19:12:39 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:39.761607430Z" level=debug msg="attach: stdout: end"
Nov 22 19:12:39 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:39.761633822Z" level=debug msg="attach: stderr: end"
Nov 22 19:12:39 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:39.761716796Z" level=debug msg="attach done"
Nov 22 19:12:49 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:49.734647938Z" level=error msg="Container failed to exit within 10 seconds of kill - trying direct SIGKILL" container=6978d7c0917634b5493c124fad2d510397685abccea54fcdbc5a49cf86831618 error="context deadline exceeded"
Nov 22 19:12:49 ip-172-31-12-189 dockerd[2383]: time="2022-11-22T19:12:49.761878423Z" level=warning msg="Logger didn't exit in time: logs may be truncated"

And, here is the stack information:

goroutine 16 [running]:
github.com/docker/docker/pkg/signal.DumpStacks({0x557e467df5c4, 0xf})
	/go/src/github.com/docker/docker/pkg/signal/trap.go:83 +0x90
github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap.func1()
	/go/src/github.com/docker/docker/daemon/debugtrap_unix.go:20 +0x7d
created by github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap
	/go/src/github.com/docker/docker/daemon/debugtrap_unix.go:18 +0xd5

goroutine 1 [chan receive, 35 minutes]:
main.(*DaemonCli).start(0xc000672630, 0xc000107a40)
	/go/src/github.com/docker/docker/cmd/dockerd/daemon.go:249 +0x130d
main.runDaemon(...)
	/go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:14
main.newDaemonCommand.func1(0xc00051adc0?, {0xc000672360?, 0x3?, 0x3?})
	/go/src/github.com/docker/docker/cmd/dockerd/docker.go:34 +0x5c
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc00051adc0, {0xc000104050, 0x3, 0x3})
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:850 +0x67c
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00051adc0)
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:958 +0x39c
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:895
main.main()
	/go/src/github.com/docker/docker/cmd/dockerd/docker.go:97 +0x15d

goroutine 14 [select]:
github.com/docker/docker/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0007398b0)
	/go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:154 +0xad
created by github.com/docker/docker/vendor/go.opencensus.io/stats/view.init.0
	/go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:32 +0x85

goroutine 42 [syscall]:
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:151 +0x2f
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
	/usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 15 [chan receive, 3 minutes]:
github.com/docker/docker/pkg/signal.Trap.func1()
	/go/src/github.com/docker/docker/pkg/signal/trap.go:38 +0x65
created by github.com/docker/docker/pkg/signal.Trap
	/go/src/github.com/docker/docker/pkg/signal/trap.go:36 +0x152

goroutine 44 [chan receive]:
github.com/docker/docker/daemon.(*Daemon).execCommandGC(0xc0000001e0)
	/go/src/github.com/docker/docker/daemon/exec.go:306 +0x65
created by github.com/docker/docker/daemon.NewDaemon
	/go/src/github.com/docker/docker/daemon/daemon.go:1145 +0x2eaa

goroutine 43 [sync.Cond.Wait, 35 minutes]:
sync.runtime_notifyListWait(0xc00004f590, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x0?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
github.com/docker/docker/daemon/stats.(*Collector).Run(0xc00004f540)
	/go/src/github.com/docker/docker/daemon/stats/collector.go:96 +0x18e
created by github.com/docker/docker/daemon.(*Daemon).newStatsCollector
	/go/src/github.com/docker/docker/daemon/stats_collector.go:24 +0x9a

goroutine 55 [IO wait, 35 minutes]:
internal/poll.runtime_pollWait(0x7f3088302788, 0x72)
	/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000142100?, 0x2?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000142100)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc000142100)
	/usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*UnixListener).accept(0x557e4510a866?)
	/usr/local/go/src/net/unixsock_posix.go:166 +0x1c
net.(*UnixListener).Accept(0xc0005be570)
	/usr/local/go/src/net/unixsock.go:260 +0x3d
net/http.(*Server).Serve(0xc0001f80e0, {0x557e47146d20, 0xc0005be570})
	/usr/local/go/src/net/http/server.go:3039 +0x385
net/http.Serve(...)
	/usr/local/go/src/net/http/server.go:2543
github.com/docker/docker/daemon.(*Daemon).listenMetricsSock.func1()
	/go/src/github.com/docker/docker/daemon/metrics_unix.go:34 +0xd4
created by github.com/docker/docker/daemon.(*Daemon).listenMetricsSock
	/go/src/github.com/docker/docker/daemon/metrics_unix.go:32 +0x14c

goroutine 56 [select, 35 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00032e840)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0x8f
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x1ca

goroutine 57 [chan receive, 35 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000440000)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x478
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x145

goroutine 59 [IO wait]:
internal/poll.runtime_pollWait(0x7f3088302698, 0x72)
	/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000142300?, 0xc000452000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000142300, {0xc000452000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000142300, {0xc000452000?, 0x557e456bd418?, 0xc00046a000?})
	/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc00012c2d0, {0xc000452000?, 0x100050020?, 0x800010601?})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0005c5560, {0xc00043e118, 0x9, 0x18?})
	/usr/local/go/src/bufio/bufio.go:236 +0x1b4
io.ReadAtLeast({0x557e4712a8d8, 0xc0005c5560}, {0xc00043e118, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:331 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:350
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader({0xc00043e118?, 0x9?, 0x557e482659e0?}, {0x557e4712a8d8?, 0xc0005c5560?})
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00043e0e0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0x95
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00046a000)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x233
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0x142f

goroutine 60 [select]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000108690, 0x1)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x115
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0005c5680)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x85
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x65
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0x163c

goroutine 61 [select, 35 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00032ec40)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0x8f
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x1ca

goroutine 62 [chan receive, 35 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0004402c0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x478
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x145

goroutine 64 [IO wait, 35 minutes]:
internal/poll.runtime_pollWait(0x7f30883025a8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000142780?, 0xc00070c000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000142780, {0xc00070c000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000142780, {0xc00070c000?, 0xc000085d10?, 0x557e44ff2b40?})
	/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc00012c338, {0xc00070c000?, 0xc000003860?, 0x800000601?})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0005c5c20, {0xc00043e2d8, 0x9, 0x557e44fc6045?})
	/usr/local/go/src/bufio/bufio.go:236 +0x1b4
io.ReadAtLeast({0x557e4712a8d8, 0xc0005c5c20}, {0xc00043e2d8, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:331 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:350
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader({0xc00043e2d8?, 0x9?, 0x7f30b0905101?}, {0x557e4712a8d8?, 0xc0005c5c20?})
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00043e2a0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0x95
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00046a1c0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x233
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0x142f

goroutine 65 [select, 35 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000108e10, 0x1)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x115
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0005c5d40)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x85
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x65
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0x163c

goroutine 66 [select, 35 minutes]:
github.com/docker/docker/libcontainerd/remote.(*client).processEventStream(0xc0001fec40, {0x557e471484a0, 0xc0007ffa40}, {0x557e467da3f8, 0xc})
	/go/src/github.com/docker/docker/libcontainerd/remote/client.go:783 +0x25e
created by github.com/docker/docker/libcontainerd/remote.NewClient
	/go/src/github.com/docker/docker/libcontainerd/remote/client.go:70 +0x33e

goroutine 67 [select, 35 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream.func5()
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:319 +0x92
created by github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:318 +0x1345

goroutine 68 [select, 35 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0005e4b00)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:318 +0x7c
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:333
github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000142c00, {0x557e4706c320?, 0xc00051e180}, 0xc00051e180?)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:871 +0xba
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x557e467cd81e?)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:736 +0x25
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0005ead80, 0xc000086e98, 0xc000086e68)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:594 +0xd3
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0005ead80, {0x557e4706c320?, 0xc00051e180?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:735 +0x11f
github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1.(*eventsSubscribeClient).Recv(0xc00047e230)
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1/events.pb.go:354 +0x4c
github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe.func1()
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:99 +0xc2
created by github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:95 +0x1ca

goroutine 45 [select]:
github.com/docker/docker/libcontainerd/remote.(*client).processEventStream(0xc0001db9d0, {0x557e471484a0, 0xc0007ffa40}, {0x557e467c97de, 0x4})
	/go/src/github.com/docker/docker/libcontainerd/remote/client.go:783 +0x25e
created by github.com/docker/docker/libcontainerd/remote.NewClient
	/go/src/github.com/docker/docker/libcontainerd/remote/client.go:70 +0x33e

goroutine 69 [select, 35 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream.func5()
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:319 +0x92
created by github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:318 +0x1345

goroutine 70 [select]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0001092c0, {0xc000125120, 0x5, 0x5})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:186 +0xa5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0001092c0, {0xc000125120?, 0xc000157f08?, 0xc000445a48?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:166 +0x147
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00051ca20, {0xc000125120?, 0xc000445ac0?, 0x557e456b6807?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:479 +0x32
io.ReadAtLeast({0x557e4712eed8, 0xc00051ca20}, {0xc000125120, 0x5, 0x5}, 0x5)
	/usr/local/go/src/io/io.go:331 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:350
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0005e4c00, {0xc000125120, 0x5, 0x5})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:463 +0xa5
github.com/docker/docker/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc000125110, 0x1000000)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:508 +0x47
github.com/docker/docker/vendor/google.golang.org/grpc.recvAndDecompress(0xc000445c88?, 0xc0005e4c00, {0x0, 0x0}, 0x1000000, 0x0, {0x0, 0x0})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:639 +0x66
github.com/docker/docker/vendor/google.golang.org/grpc.recv(0xc0001db7a0?, {0x7f3088136468, 0x557e482964e8}, 0x0?, {0x0?, 0x0?}, {0x557e4706c320, 0xc001276f00}, 0x557e44fc6045?, 0x0, ...)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:707 +0x6e
github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000142c80, {0x557e4706c320?, 0xc001276f00}, 0xc001276f00?)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:885 +0x2b0
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc00051e4e0?)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:736 +0x25
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0005eafc0, 0xc000445e98, 0xc000445e68)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:590 +0x2f6
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0005eafc0, {0x557e4706c320?, 0xc001276f00?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:735 +0x11f
github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1.(*eventsSubscribeClient).Recv(0xc00047e330)
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1/events.pb.go:354 +0x4c
github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe.func1()
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:99 +0xc2
created by github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:95 +0x1ca

goroutine 46 [select]:
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).watchLoop(0xc00015b300)
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/store.go:430 +0x86
created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startWatch
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/store.go:447 +0xef

goroutine 653 [select, 3 minutes]:
github.com/docker/docker/container.(*State).Wait.func1()
	/go/src/github.com/docker/docker/container/state.go:213 +0xc7
created by github.com/docker/docker/container.(*State).Wait
	/go/src/github.com/docker/docker/container/state.go:212 +0x2bb

goroutine 687 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f30881a1ac0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000c7f600?, 0xc0009fc000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000c7f600, {0xc0009fc000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000c7f600, {0xc0009fc000?, 0x557e45097b66?, 0xc000e03958?})
	/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc00012ce70, {0xc0009fc000?, 0x0?, 0x0?})
	/usr/local/go/src/net/net.go:183 +0x45
net/http.(*connReader).Read(0xc0000dfc80, {0xc0009fc000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:780 +0x16d
bufio.(*Reader).fill(0xc0009e9260)
	/usr/local/go/src/bufio/bufio.go:106 +0x103
bufio.(*Reader).Peek(0xc0009e9260, 0x4)
	/usr/local/go/src/bufio/bufio.go:144 +0x5d
net/http.(*conn).readRequest(0xc000a366e0, {0x557e471484a0, 0xc000a93200})
	/usr/local/go/src/net/http/server.go:985 +0x1f8
net/http.(*conn).serve(0xc000a366e0, {0x557e47148548, 0xc000cb8390})
	/usr/local/go/src/net/http/server.go:1891 +0x32b
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3071 +0x4db

goroutine 703 [semacquire]:
sync.runtime_SemacquireMutex(0xc00091cc60?, 0x45?, 0x0?)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0xc000973888)
	/usr/local/go/src/sync/mutex.go:162 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0x557e44fc63a7?)
	/usr/local/go/src/sync/rwmutex.go:139 +0x36
github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).Close(0xc0009737a0)
	/go/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:389 +0xca
github.com/docker/docker/daemon/logger/fluentd.(*fluentd).Close(0x0?)
	/go/src/github.com/docker/docker/daemon/logger/fluentd/fluentd.go:130 +0x1d
github.com/docker/docker/daemon/logger/loggerutils/cache.(*loggerWithCache).Close(0xc000aef900)
	/go/src/github.com/docker/docker/daemon/logger/loggerutils/cache/local_cache.go:92 +0x2d
github.com/docker/docker/container.(*Container).Reset(0xc0003c4280, 0x10?)
	/go/src/github.com/docker/docker/container/monitor.go:45 +0x4ba
github.com/docker/docker/daemon.(*Daemon).handleContainerExit(0xc0000001e0, 0xc0003c4280, 0xc00091d360)
	/go/src/github.com/docker/docker/daemon/monitor.go:40 +0x2b7
github.com/docker/docker/daemon.(*Daemon).ProcessEvent(0xc0000001e0, {0xc000cbe940, 0x40}, {0x557e467c95a2, 0x4}, {{0xc000cbe940, 0x40}, {0xc000cbe980, 0x40}, 0xf0d, ...})
	/go/src/github.com/docker/docker/daemon/monitor.go:129 +0x5c5
github.com/docker/docker/libcontainerd/remote.(*client).processEvent.func1()
	/go/src/github.com/docker/docker/libcontainerd/remote/client.go:678 +0xbc
github.com/docker/docker/libcontainerd/queue.(*Queue).Append.func1()
	/go/src/github.com/docker/docker/libcontainerd/queue/queue.go:28 +0x63
created by github.com/docker/docker/libcontainerd/queue.(*Queue).Append
	/go/src/github.com/docker/docker/libcontainerd/queue/queue.go:24 +0x205

goroutine 652 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f30881a1ca0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000143200?, 0xc000840551?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000143200, {0xc000840551, 0x1, 0x1})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000143200, {0xc000840551?, 0xc000c319d8?, 0xc000790768?})
	/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000010cc8, {0xc000840551?, 0xc000f05720?, 0x10?})
	/usr/local/go/src/net/net.go:183 +0x45
net/http.(*connReader).backgroundRead(0xc000840540)
	/usr/local/go/src/net/http/server.go:672 +0x3f
created by net/http.(*connReader).startBackgroundRead
	/usr/local/go/src/net/http/server.go:668 +0xca

goroutine 726 [semacquire]:
sync.runtime_SemacquireMutex(0x300000002?, 0xa0?, 0xc0004349b8?)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0xc000143a00)
	/usr/local/go/src/sync/mutex.go:162 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
github.com/docker/docker/container.(*State).GetPID(0xf0?)
	/go/src/github.com/docker/docker/container/state.go:248 +0x36
github.com/docker/docker/daemon.killProcessDirectly(0xc0003c4280)
	/go/src/github.com/docker/docker/daemon/container_operations_unix.go:339 +0x3d
github.com/docker/docker/daemon.(*Daemon).Kill(0xc000b10770?, 0xc0003c4280)
	/go/src/github.com/docker/docker/daemon/kill.go:158 +0x259
github.com/docker/docker/daemon.(*Daemon).containerStop(0xc0000001e0, 0xc0003c4280, 0xa)
	/go/src/github.com/docker/docker/daemon/stop.go:88 +0x3e9
github.com/docker/docker/daemon.(*Daemon).ContainerStop(0x557e46de8820?, {0xc0009b4c17, 0x5}, 0x0)
	/go/src/github.com/docker/docker/daemon/stop.go:33 +0xb5
github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersStop(0xc00095de00, {0xb57853b1990419f?, 0xc0009d13d0?}, {0x557e47146f00, 0xc000f9f0a0}, 0xc000936400, 0xc0000d5b80?)
	/go/src/github.com/docker/docker/api/server/router/container/container_routes.go:233 +0x118
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1({0x557e47148548, 0xc000f79080}, {0x557e47146f00?, 0xc000f9f0a0?}, 0x557e46d02a40?, 0xc000d0dff0?)
	/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x15b
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1({0x557e47148548, 0xc000f78ff0}, {0x557e47146f00, 0xc000f9f0a0}, 0x0?, 0x7f30b09055b8?)
	/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x4d7
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1({0x557e47148548, 0xc000f78ff0}, {0x557e47146f00?, 0xc000f9f0a0}, 0xc000936400, 0x2?)
	/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x649
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1({0x557e47148548, 0xc000f78ff0}, {0x557e47146f00, 0xc000f9f0a0}, 0xc000936400, 0xc000d0def0?)
	/go/src/github.com/docker/docker/api/server/middleware/debug.go:25 +0x633
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1({0x557e47146f00, 0xc000f9f0a0}, 0xc000936300)
	/go/src/github.com/docker/docker/api/server/server.go:141 +0x2fa
net/http.HandlerFunc.ServeHTTP(0xc000936000?, {0x557e47146f00?, 0xc000f9f0a0?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2084 +0x2f
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00066f680, {0x557e47146f00, 0xc000f9f0a0}, 0xc0009ff600)
	/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0x1cf
net/http.serverHandler.ServeHTTP({0xc000f78e70?}, {0x557e47146f00, 0xc000f9f0a0}, 0xc0009ff600)
	/usr/local/go/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc000762b40, {0x557e47148548, 0xc000cb8390})
	/usr/local/go/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3071 +0x4db

goroutine 88 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f30883024b8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc00074d200?, 0x18?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc00074d200)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc00074d200)
	/usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*UnixListener).accept(0x557e44ffac00?)
	/usr/local/go/src/net/unixsock_posix.go:166 +0x1c
net.(*UnixListener).Accept(0xc0008009c0)
	/usr/local/go/src/net/unixsock.go:260 +0x3d
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).acceptClientConnections(0xc00015b300, {0xc00012bdd0, 0x2c}, {0x557e47146d20, 0xc0008009c0})
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox_externalkey_unix.go:144 +0x5b
created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startExternalKeyListener
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox_externalkey_unix.go:138 +0x31e

goroutine 87 [select, 35 minutes]:
github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay.(*driver).peerOpRoutine(0x0?, {0x557e471484a0, 0xc00077a280}, 0xc0007265a0)
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay/peerdb.go:279 +0xe8
created by github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay.Init
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay/overlay.go:77 +0x218

goroutine 650 [chan receive, 3 minutes]:
github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersWait(0xc00095de00, {0x557e47148548, 0xc000728720}, {0x557e47146f00?, 0xc00043fa40?}, 0xc000819500, 0xc00095fa20?)
	/go/src/github.com/docker/docker/api/server/router/container/container_routes.go:365 +0x325
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1({0x557e47148548, 0xc000728720}, {0x557e47146f00?, 0xc00043fa40?}, 0x557e46d02a40?, 0xc000b2d840?)
	/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x15b
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1({0x557e47148548, 0xc000728690}, {0x557e47146f00, 0xc00043fa40}, 0x0?, 0x7f30b0905108?)
	/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x4d7
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1({0x557e47148548, 0xc000728690}, {0x557e47146f00?, 0xc00043fa40}, 0xc000819500, 0x2?)
	/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x649
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1({0x557e47148548, 0xc000728690}, {0x557e47146f00, 0xc00043fa40}, 0xc000819500, 0xc000b2d700?)
	/go/src/github.com/docker/docker/api/server/middleware/debug.go:25 +0x633
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1({0x557e47146f00, 0xc00043fa40}, 0xc000819400)
	/go/src/github.com/docker/docker/api/server/server.go:141 +0x2fa
net/http.HandlerFunc.ServeHTTP(0xc000819300?, {0x557e47146f00?, 0xc00043fa40?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2084 +0x2f
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00066f680, {0x557e47146f00, 0xc00043fa40}, 0xc000819200)
	/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0x1cf
net/http.serverHandler.ServeHTTP({0xc000840540?}, {0x557e47146f00, 0xc00043fa40}, 0xc000819200)
	/usr/local/go/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc00084b400, {0x557e47148548, 0xc000cb8390})
	/usr/local/go/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3071 +0x4db

goroutine 705 [semacquire]:
sync.runtime_Semacquire(0x557e4712b558?)
	/usr/local/go/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000fb00c0?)
	/usr/local/go/src/sync/waitgroup.go:136 +0x52
github.com/docker/docker/daemon/logger.(*Copier).Wait(...)
	/go/src/github.com/docker/docker/daemon/logger/copier.go:183
github.com/docker/docker/container.(*Container).Reset.func1()
	/go/src/github.com/docker/docker/container/monitor.go:33 +0x33
created by github.com/docker/docker/container.(*Container).Reset
	/go/src/github.com/docker/docker/container/monitor.go:32 +0x398

goroutine 688 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000a93710, 0x11)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x557e44fcfeaf?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
github.com/docker/docker/daemon/logger.(*messageRing).Dequeue(0xc000a936c0)
	/go/src/github.com/docker/docker/daemon/logger/ring.go:193 +0x4b
github.com/docker/docker/daemon/logger.(*RingLogger).run(0xc00066d0e0)
	/go/src/github.com/docker/docker/daemon/logger/ring.go:125 +0x55
created by github.com/docker/docker/daemon/logger.newRingLogger
	/go/src/github.com/docker/docker/daemon/logger/ring.go:45 +0x1d8

goroutine 119 [chan receive, 35 minutes]:
github.com/docker/docker/api/server.(*Server).serveAPI(0xc000738dc0)
	/go/src/github.com/docker/docker/api/server/server.go:94 +0x18a
github.com/docker/docker/api/server.(*Server).Wait(0x557e4510d92a?, 0x0?)
	/go/src/github.com/docker/docker/api/server/server.go:199 +0x25
created by main.(*DaemonCli).start
	/go/src/github.com/docker/docker/cmd/dockerd/daemon.go:242 +0x12ea

goroutine 117 [chan receive, 35 minutes]:
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop.func2()
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:76 +0x2c
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:75 +0x96

goroutine 118 [chan receive, 5 minutes]:
main.(*DaemonCli).setupConfigReloadTrap.func1()
	/go/src/github.com/docker/docker/cmd/dockerd/daemon_unix.go:84 +0x4d
created by main.(*DaemonCli).setupConfigReloadTrap
	/go/src/github.com/docker/docker/cmd/dockerd/daemon_unix.go:83 +0xb9

goroutine 166 [sync.Cond.Wait, 35 minutes]:
sync.runtime_notifyListWait(0xc00095dcd0, 0x0)
	/usr/local/go/src/runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0xc00047f830?)
	/usr/local/go/src/sync/cond.go:56 +0x8c
github.com/docker/docker/vendor/github.com/moby/buildkit/util/cond.(*StatefulCond).Wait(0xc0009b2e40)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/cond/cond.go:28 +0x6c
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop(0xc0002795e0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:101 +0x1e5
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.newScheduler
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:35 +0x205

goroutine 167 [select, 35 minutes]:
github.com/docker/docker/daemon.(*Daemon).ProcessClusterNotifications(0x0?, {0x557e471484a0, 0xc0007ffa40}, 0xc00097f920)
	/go/src/github.com/docker/docker/daemon/events.go:126 +0xad
created by main.(*DaemonCli).start
	/go/src/github.com/docker/docker/cmd/dockerd/daemon.go:234 +0x1250

goroutine 168 [IO wait]:
internal/poll.runtime_pollWait(0x7f3088302878, 0x72)
	/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0005d9a00?, 0x18?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc0005d9a00)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc0005d9a00)
	/usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*UnixListener).accept(0x557e45021cf0?)
	/usr/local/go/src/net/unixsock_posix.go:166 +0x1c
net.(*UnixListener).Accept(0xc000673650)
	/usr/local/go/src/net/unixsock.go:260 +0x3d
net/http.(*Server).Serve(0xc0001f8000, {0x557e47146d20, 0xc000673650})
	/usr/local/go/src/net/http/server.go:3039 +0x385
github.com/docker/docker/api/server.(*HTTPServer).Serve(...)
	/go/src/github.com/docker/docker/api/server/server.go:112
github.com/docker/docker/api/server.(*Server).serveAPI.func1(0xc000875050)
	/go/src/github.com/docker/docker/api/server/server.go:86 +0x96
created by github.com/docker/docker/api/server.(*Server).serveAPI
	/go/src/github.com/docker/docker/api/server/server.go:83 +0x52

goroutine 397 [select]:
github.com/docker/docker/vendor/github.com/docker/libnetwork/osl.removeUnusedPaths()
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/osl/namespace_linux.go:94 +0x105
created by github.com/docker/docker/vendor/github.com/docker/libnetwork/osl.createBasePath
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/osl/namespace_linux.go:79 +0x85

goroutine 689 [select]:
github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).connectWithRetry(0xc0009737a0, {0x557e471484d8, 0xc00012e000})
	/go/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:470 +0x17a
github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).write.func2(0xc0009737a0, {0x557e471484d8, 0xc00012e000})
	/go/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:563 +0x9d
github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).write(0xc0009737a0, {0x557e471484d8?, 0xc00012e000?}, 0xc0007bff80)
	/go/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:567 +0x65
github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).writeWithRetry(0xc0009737a0, {0x557e471484d8, 0xc00012e000}, 0xc0007bff20?)
	/go/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:537 +0x5d
github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).postRawData(0xc0009737a0?, 0xc0005c9060?)
	/go/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:292 +0x5b
github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).EncodeAndPostData(0x557e46de7e60?, {0xc0005c9060?, 0x557e467c8a56?}, {0x3?, 0xc000d04e10?, 0x0?}, {0x557e46de7e60, 0xc0007bff20})
	/go/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:275 +0xb5
github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).PostWithTime(0xc0009737a0, {0xc0005c9060?, 0x557e467c8a56?}, {0x3?, 0xc000d96c78?, 0x0?}, {0x557e46de8820?, 0xc0007bfef0})
	/go/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:266 +0x399
github.com/docker/docker/daemon/logger/fluentd.(*fluentd).Log(0xc000a93640, 0xc000cbbf80)
	/go/src/github.com/docker/docker/daemon/logger/fluentd/fluentd.go:126 +0x4f4
github.com/docker/docker/daemon/logger/loggerutils/cache.(*loggerWithCache).Log(0xc000aef900, 0xc000cbbf80)
	/go/src/github.com/docker/docker/daemon/logger/loggerutils/cache/local_cache.go:77 +0x1b5
github.com/docker/docker/daemon/logger.(*Copier).copySrc(0xc000a93840, {0x557e467cd201, 0x6}, {0x7f308804d768, 0xc000a93740})
	/go/src/github.com/docker/docker/daemon/logger/copier.go:128 +0xf0d
created by github.com/docker/docker/daemon/logger.(*Copier).Run
	/go/src/github.com/docker/docker/daemon/logger/copier.go:48 +0x6c

[abicky]

Although I'm not familiar with the code, according to the stacks, the following happened:

1. goroutine 689 acquired the lock of muconn
   • https://github.com/moby/moby/blob/v20.10.21/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go#L559-L564
2. goroutine 689 tried to connect to the fluentd up to math.MaxInt32 times
   • https://github.com/moby/moby/blob/v20.10.21/daemon/logger/fluentd/fluentd.go#L46-L48
3. goroutine 703 called handleContainerExit and acquired the lock of the container
   • https://github.com/moby/moby/blob/v20.10.21/daemon/monitor.go#L28
4. goroutine 703 tried to close the connection and waited for the lock of muconn to be released, that is, it waited for goroutine 689 to connect to the fluentd
   • https://github.com/moby/moby/blob/v20.10.21/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go#L389
5. goroutine 726 tried to kill the container and waited for the lock of the container to be released
   • https://github.com/moby/moby/blob/v20.10.21/daemon/container_operations_unix.go#L339

[corhere]

• Container hangs when using fluentd logger #40063

The fix was backported to v20.10.13, though it only takes effect when the log option fluentd-async-connect=true is set. Does setting that option resolve the issue for you @abicky?

[sam-thibault]

I don't see any recent activity on this issue. I will close it as stale.