Docker daemon hangs #1681
KVM on CentOS 7 (3.10.0-327.36.3.el7.x86_64)
Docker daemon should be responsible for commands such as ps, run, stop, etc..
Docker daemon hangs on commands such as ps, run, stop, etc.
I cannot find exact reproduction step yet (I'm making simplified version of my real service, but yet success to reproduce the issue). However, it occurs almost every time when I repeat creating and removing containers a lot (more than about 1,000 containers, but sometimes with fewer containers. Each container lives 1s ~ 120s).
Stacktrace of dockerd : sigusr1.txt
I first tried with coreos stable(1185.3.0) and suffered from another issue, and moved to coreos alpha (1235.0.0) and then this issue is occurred. Here is the error log what I got in stable channel.
The text was updated successfully, but these errors were encountered:
I had tested the same service with CentOS (3.10.0-327.36.3.el7.x86_64) + Docker 1.12.3, which is the same version of current CoreOS alpha channel, with zfs storage driver (because overlay driver simply does not work) and found that the issue is not reproduced (~7 days with no hang). Therefore, I posted the issue here rather than github.com/docker/docker.
The issue still exists after updating coreOS to 1262.0.0 (alpha).
We are getting hans on 1235.0.0 - 1235.12.0 which encompasses coreos docker versions 1.12.3 - 1.12.6
here is stack trace from 1235.12.0 https://gist.github.com/someword/6ca8abc4782dcc6c94c8803cab32a93a
I had to pull the stack trace from journalctl. Not sure if the formatting is in the desired format.
journalctl -u docker --no-pager --since="17:53:48" --until="17:54:00" -o cat >/var/tmp/docker-usr1.txt
Os and docker details below
core@ip-10-20-2-103 ~ $ cat /etc/os-release