Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
docker timeouts when waiting for mkfs to finish on devicemapper storage driver device #16653
By default on RHEL 7.1 docker creates sparse file one upon start (systemctl start docker). That file is used as a loopback-device (seemingly using dmsetup syscall eqivalents), then issues
This command may take several minutes to complete. Meanwhile the
Remove the backing files. I am not sure about the order of dmsetup lines here.
For some time, we can see the mkfs being performed
But the user-space process disappears before the kernel-space process, while the queue is still quite full.
The result is
It seems mkfs wrote about at least 150MB of data:
subsequent mkfs executed on the device exits fast, I think this is caused by the fact it is thin-provisioned.
I can't see any meaningfull error message in the logs that would help me debugging this problem.
Note that when one tries to start the docker again, docker will re-create the sparse file so the start is likely to fail again even though the FS is created fine every time, because this is done asychronously
Docker started on system no matter how much loaded it is. This would probably need checking of the mkfs progress.
Maybe the -D mkfs option may help.
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues
BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
This primarily happens because your underlying storage is slow. And recently we had increased the default size of base image to 100G (from 10G) so on slower storage problem became worse and it started to run into systemd timeout.
Quick work around are following.
In the long run, I think we should revert the patch to reduce the base image size to say 20G and allow a docker create/run option which can specify the size of container at creation time. (Say --storage-opt dm.size=100G). That way one can start with a small base size but specify a bigger size for specific containers. This might still run into build issues though.
@jaryn docker-storage-setup utility can easily setup lvm thin pool for you.
We also ship it as part of docker package in fedora/rhel/centos.