New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker rmi can not clean up as fast as new images are created #11053

Closed
tfoote opened this Issue Feb 26, 2015 · 8 comments

Comments

Projects
None yet
6 participants
@tfoote

tfoote commented Feb 26, 2015

I have am running a continuous integration and build server using docker as isolation. We are heavily leverating docker isolation for dependency checking and repeatilble builds. However due to the variety of packages which we are building relatively quickly run out of disk space.

To prevent running out of disk space we have developed a script which cleans up old images and containers. There's discussion related to that here: #928 However, having worked around those issues we've discovered that on a loaded system docker cannot clean images faster than it can create them.

On one of the loaded systems I observed a single rmi of a moderately small image taking over 15 minutes. This results in the script which is iterating through the list of images previously determined to be safe to be removed clearing disk space more slowly than new images are being created. As a result when under heavy load the machines regularly have to take themselves offline due to lack of disk space and wait for the cleanup script to catch up. In that same amount of time often 3-4 new similar containers with several layers of images have been built by other processes interacting with docker.

This is a quad core/ 16GB specifically an EC2 xlarge with 200GB of EBS optimized storage. (Specifics below in the docker info)

I understand that I'm heavily loading the system and that things are expected to slow down. But I believe that the asymmetry of slowdown between the build and rmi commands is undesired, and I expect that rmi should operate faster as it's just needing to deindex data instead of creating an inserting new data.

To reproduce this run 3-4 different builds generating similar but not identical containers simultaneously. As disk space starts getting full, (~175/200Gb) try to rmi the outdated images. We're using this python script: https://github.com/ros-infrastructure/buildfarm_deployment/blob/master/slave/slave_files/files/home/jenkins-slave/cleanup_docker_images.py But simply calling docker rmi on the command line can reproduce.

Timing for removing an image under load:

$ time docker rmi sourcedeb__indigo_ubuntu_saucy_ff
Untagged: sourcedeb__indigo_ubuntu_saucy_ff:latest
Deleted: e2e6fb5705784775db1afa6f022b0e557cd7c657ea6f15cef8419300d4e5e68a
Deleted: 12a1bf62376395c013a98c69485eedd5848b79dfa8a9cf31471bf660eddb4a8e
Deleted: 8c9987e3ae342be0ff925c68108285afcf2ffd70fdae8c041da3fecd6ad3598e
Deleted: e1c442d9036f4efa0124893872dc2c6b9294d909c8016fa42f68ab2563330e21
Deleted: 2262069b504da149a0352c2b15001687c288354abbbb3857e22e1f83f6e94fd8

real    16m19.530s
user    0m0.024s
sys 0m0.001s

It's not a large image(info on that image from docker images)

sourcedeb__indigo_ubuntu_saucy_ff                                                       latest              e2e6fb570578        41 hours ago         516.3 MB

Removing a containers is reasonably quick

$ time docker rm 69cb58ff41f7
69cb58ff41f7

real    0m4.625s
user    0m0.008s
sys 0m0.008s

Listing images is also slower than I would expect

$ time docker images
[[SNIP 700 images]]
real    0m22.014s
user    0m0.047s
sys 0m0.033s

The standard docker information

root@ip-172-31-14-178:~# docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
root@ip-172-31-14-178:~# docker info
Containers: 9
Images: 95
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 113
Execution Driver: native-0.2
Kernel Version: 3.13.0-44-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 4
Total Memory: 14.69 GiB
Name: ip-172-31-14-178
ID: 6KW6:XMAA:BLMZ:F5U2:QMGY:JNZR:FCZ5:FVDU:GJUS:KSHQ:FSR3:AO62
WARNING: No swap limit support
root@ip-172-31-14-178:~# uname -a
Linux ip-172-31-14-178 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

@tfoote tfoote referenced this issue Feb 26, 2015

Closed

docker cleanup issues #50

3 of 5 tasks complete
@ruffsl

This comment has been minimized.

ruffsl commented Jun 29, 2015

+1
I am also witnessing poor performance in listing and deleting containers

Listing images:

$ time docker images
[[SNIP 6831 images]]
real    0m31.818s
user    0m0.061s
sys     0m0.039s

Removing a containers:

$ time docker rmi 916bac8f5800
Deleted: 916bac8f5800
real    0m22.520s
user    0m0.016s
sys     0m0.008s

The standard docker information:

$ docker info
Containers: 4
Images: 6831
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 6860
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-48-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 15.67 GiB
Name: ip-172-31-10-44
ID: VO2N:RR5Z:MOYG:SQDU:EX5F:R6UW:FXEQ:XWDT:W2EL:WX5K:UCBF:RZDK
WARNING: No swap limit support
$ docker version 
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): linux/amd64
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64

Is there perhaps a best approach in handling and working with large numbers of images?

@andyp1per

This comment has been minimized.

andyp1per commented Aug 10, 2015

+1 docker 1.7.1 on overlay. I was hoping overlay would make things much better but it does not.

@mbrewster

This comment has been minimized.

mbrewster commented Sep 17, 2015

+1 on docker 1.8.2

@GordonTheTurtle

This comment has been minimized.

GordonTheTurtle commented Sep 25, 2015

USER POLL

The best way to get notified when there are changes in this discussion is by clicking the Subscribe button in the top right.

The people listed below have appreciated your meaningfull discussion with a random +1:

@hai-ld
@termie

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Oct 14, 2015

Looks like this is now being worked on in #16771

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Dec 9, 2015

@tfoote is this still an issue, now that #16890 has been merged (docker 1.9.x)?

@tfoote

This comment has been minimized.

tfoote commented Jan 13, 2016

@thaJeztah It's fixed with 1.9.1 I can rmi a similar image under similar load in under 1 second.

@tfoote tfoote closed this Jan 13, 2016

@thaJeztah thaJeztah added this to the 1.10.0 milestone Jan 13, 2016

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jan 13, 2016

Thanks for testing @tfoote, happy to hear it's resolved!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment