Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Docker daemon hangs #1681
KVM on CentOS 7 (3.10.0-327.36.3.el7.x86_64)
Docker daemon should be responsible for commands such as ps, run, stop, etc..
Docker daemon hangs on commands such as ps, run, stop, etc.
I cannot find exact reproduction step yet (I'm making simplified version of my real service, but yet success to reproduce the issue). However, it occurs almost every time when I repeat creating and removing containers a lot (more than about 1,000 containers, but sometimes with fewer containers. Each container lives 1s ~ 120s).
Stacktrace of dockerd : sigusr1.txt
I first tried with coreos stable(1185.3.0) and suffered from another issue, and moved to coreos alpha (1235.0.0) and then this issue is occurred. Here is the error log what I got in stable channel.
I had tested the same service with CentOS (3.10.0-327.36.3.el7.x86_64) + Docker 1.12.3, which is the same version of current CoreOS alpha channel, with zfs storage driver (because overlay driver simply does not work) and found that the issue is not reproduced (~7 days with no hang). Therefore, I posted the issue here rather than github.com/docker/docker.
The issue still exists after updating coreOS to 1262.0.0 (alpha).
modified the milestones:
Container Linux Stable 1235.7.0,
CoreOS Alpha 1300.0.0
Jan 24, 2017
We are getting hans on 1235.0.0 - 1235.12.0 which encompasses coreos docker versions 1.12.3 - 1.12.6
here is stack trace from 1235.12.0 https://gist.github.com/someword/6ca8abc4782dcc6c94c8803cab32a93a
I had to pull the stack trace from journalctl. Not sure if the formatting is in the desired format.
journalctl -u docker --no-pager --since="17:53:48" --until="17:54:00" -o cat >/var/tmp/docker-usr1.txt
Os and docker details below
core@ip-10-20-2-103 ~ $ cat /etc/os-release