journald restart crashes containerd

[x1022as]

This is closely related to moby/moby#19728 : journald restart crashes Docker

daemon and containerd's stderr point to the same socket if their logs are managed by journald:

# ls /proc/31602//fd/2 -al
lrwx------ 1 root root 64 Feb 28 15:47 /proc/31602//fd/2 -> socket:[6992842]
# ls /proc/31609/fd/2 -al
lrwx------ 1 root root 64 Feb 28 15:50 /proc/31609/fd/2 -> socket:[6992842]

restart journald service, daemon will ignore SIGPIPE and some docker operation will work properly, while all docker logs are missing, because it's trying to write its log into a broken pipe. Containerd for now still get SIGPIPE, and may get killed. The wired thing is that docker did not start a new containerd.

# systemctl restart systemd-journald.service
# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
# ps -ef | grep docker
root      9486  4081  0 16:06 pts/2    00:00:00 grep --color=auto docker
root     31602     1  0 15:47 ?        00:00:00 /usr/bin/docker daemon -D --live-restore
root     31609 31602  0 15:47 ?        00:00:00 docker-containerd -l /var/run/docker/libcontainerd/docker-containerd.sock --runtime docker-runc --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --debug --metrics-interval=0
# docker run -tid busybox sleep 10000
afdccea8dfb5483ed082547c97293f33b652fc1cacdee23247642b56b2bcee24
# docker rm -f afdccea8dfb5483ed082547c97293f33b652fc1cacdee23247642b56b2bcee24

after journald restart, docker and containerd are still there and docker run works fine. but docker rm will block forever. ps in another bash, I get this:

# ps -ef | grep docker
root     11306  4081  0 16:09 pts/2    00:00:00 docker rm -f afdccea8dfb5483ed082547c97293f33b652fc1cacdee23247642b56b2bcee24
root     11376 19869  0 16:09 pts/1    00:00:00 grep --color=auto docker
root     31602     1  0 15:47 ?        00:00:00 /usr/bin/docker daemon -D --live-restore

containerd got killed, and never start again. As all logs are missing, it will be hard to debug this.

ps. docker version is 1.11.2 but have moby/moby#22460 : Ignore SIGPIPE events backported.

---

maybe containerd need to ignore SIGPIPE too. but missing all the docker logs is kind of severe problem.

not sure if there is a better way for docker and containerd to deal with these broken pipe errors?

ping @crosbymichael @LK4D4 @jwhonce @coolljt0725

[ys-hwang]

We've also got this issue.
Like @x1022as said above, when journald restarted, containerd died, and docker did not start a new containerd.
We hope that this issue will resolve asap. Thanks.

Our docker version is 17.03.0-ce.

[hqhq]

Fixed by: #930