Docker RMI spins 100% CPU and takes a very long time to complete #3470

vincentwoo · 2014-01-06T22:25:58Z

REPOSITORY                TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
vincentwoo/coderpad-tty   latest              6c906892484f        28 minutes ago      2.577 GB
<none>                    <none>              9fa227837559        About an hour ago   1.859 GB
<none>                    <none>              fb8dc316d058        About an hour ago   769.1 MB
<none>                    <none>              3517a6e38005        2 hours ago         527.8 MB
<none>                    <none>              1b705ef25f73        18 hours ago        2.592 GB
<none>                    <none>              b6a6d564c3b2        21 hours ago        527.5 MB
<none>                    <none>              cb13170d11ee        3 weeks ago         2.553 GB
ubuntu                    12.04               8dbd9e392a96        9 months ago        128 MB

I have been watching $ docker rmi cb13170d11ee go for 5 minutes now. My laptop is getting very hot.

The text was updated successfully, but these errors were encountered:

vincentwoo · 2014-01-06T22:26:24Z

Ah, this just happened:

Error: Conflict, cb13170d11ee wasn't deleted
2014/01/06 22:22:25 Error: failed to remove one or more images

crosbymichael · 2014-01-06T22:40:28Z

ping @mzdaniel

MichaelSp · 2014-01-07T12:26:06Z

same issue here
also the docker images command takes very long:

$ time docker images
REPOSITORY               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
<none>                   <none>              82b6a8b6928a        About an hour ago   1.06 GB
... etc ...
0.00user 0.00system 0:07.22elapsed 0%CPU (0avgtext+0avgdata 4400maxresident)k

7 secs. I guess that should be quicker.
And than:

$ time docker rmi 82b6a8b6928a
^CCommand exited with non-zero status 2
0.00user 0.03system 34:03.39elapsed 0%CPU (0avgtext+0avgdata 3924maxresident)k

Aborted after 30 mins

crosbymichael · 2014-01-08T18:47:42Z

This has been fixed in 0.7.4

vincentwoo · 2014-01-09T01:30:00Z

@crosbymichael are you sure?

vagrant@precise64:~$ docker rmi cb13170d11ee
Error: Conflict, cb13170d11ee wasn't deleted
2014/01/09 01:29:15 Error: failed to remove one or more images
vagrant@precise64:~$ docker version
Client version: 0.7.4
Go version (client): go1.2
Git commit (client): 010d74e
Server version: 0.7.4
Git commit (server): 010d74e
Go version (server): go1.2
Last stable version: 0.7.4
vagrant@precise64:~$

Do I need to wipe docker's directory first? Do you mean it is fixed going forward for images made with 0.7.4?

crosbymichael · 2014-01-09T01:31:30Z

@vincentwoo I thought that the issue here was that rmi takes a long time to run, not that your getting a conflict?

vincentwoo · 2014-01-09T01:32:18Z

Ah, I see. Yeah, the conflict was from a dead container that needed to be removed. I humbly suggest that the error message say as much, or point out containers causing conflicts.

crosbymichael · 2014-01-09T01:33:16Z

@vincentwoo thanks for following up

arthur-debert · 2014-01-24T00:46:29Z

I see the same issue (taking a very long time, saying there are conflicts, fails to remove) on docker 0.7.6. Any info I could provide to help repro this?

bmurphy1976 · 2014-02-03T19:30:17Z

I'm still experiencing this problem:

root@dc461a2ed605:~# docker -v
Docker version 0.7.6, build bc3b2ec
root@dc461a2ed605:~# time docker rmi e64c97386d66
Error: Conflict, e64c97386d66 wasn't deleted
2014/02/03 19:27:01 Error: failed to remove one or more images

real    2m17.067s
user    0m0.016s
sys 0m0.000s
root@dc461a2ed605:~#

I have some containers that I can't delete and I suspect this is aggravating the situation:

root@dc461a2ed605:~# docker rm 0304313e8efb
Error: Cannot destroy container 0304313e8efb: Driver aufs failed to remove root filesystem 0304313e8efbc3a5e5448f37a796aafad6c66b0f1caa2ac7e06bd363bdd86cd9: device or resource busy
2014/02/03 19:29:41 Error: failed to remove one or more containers

socketbind · 2014-02-20T21:57:02Z

Same issue on Ubuntu 13.10 64 bit.

$ docker version
Client version: 0.8.1
Go version (client): go1.2
Git commit (client): a1598d1
Server version: 0.8.1
Git commit (server): a1598d1
Go version (server): go1.2
Last stable version: 0.8.1

$ docker info
Containers: 369
Images: 348
Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 1086
WARNING: No swap limit support

burke · 2014-04-15T22:15:45Z

Same here. rmi takes an unjustifiably long time on all docker versions, including 0.10. (ubuntu 12.04)

vieux · 2014-04-16T16:56:29Z

Can you paste use
docker version and docker info ?

burke · 2014-04-16T17:11:57Z

deploy@borg0001:~$ docker info
Containers: 62
Images: 213
Storage Driver: aufs
 Root Dir: /u/lib/docker/aufs
 Dirs: 337
Execution Driver: native-0.1
Kernel Version: 3.8.0-35-generic
WARNING: No swap limit support
deploy@borg0001:~$ docker version
Client version: 0.10.0
Client API version: 1.10
Go version (client): go1.2.1
Git commit (client): dc9c28f
Server version: 0.10.0
Server API version: 1.10
Git commit (server): dc9c28f
Go version (server): go1.2.1
Last stable version: 0.10.0

pnasrat · 2014-04-16T17:16:08Z

I'll see if I can replicate and profile.

burke · 2014-04-16T17:18:12Z

Thanks. I did a strace and noticed a LOT of activity reading from /var/lib/docker/graph/*. It seems to be walking the graph many times more than is reasonable. I looked through the code and it wasn't immediately obvious why, to me, but I only spent 5-10 minutes with it.

pnasrat · 2014-04-16T17:25:35Z

Yeah it's probably similar to what I optimized in the restore path.

I've tried with:

Containers: 4
Images: 1969
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 2111
Execution Driver: native-0.1
Kernel Version: 3.13.0-0.bpo.1-amd64
Debug mode (server): true
Debug mode (client): false
Fds: 11
Goroutines: 12
EventsListeners: 0
Init Path: /home/pnasrat/docker/bundles/0.10.0-dev/binary/docker-0.10.0-dev
WARNING: No memory limit support
WARNING: No swap limit support
[pnasrat@docker-playground:~/docker] master+ ± time docker rmi 2e5d58dfeaab                                                                                              
Untagged: demo:1397162605-36
Deleted: 2e5d58dfeaabc9584ff5a2327709258bd2da948545055fc0662341e119f6020f

real    0m1.110s
user    0m0.004s
sys     0m0.004s

As I can't reproduce with head and many more images, would you be willing to profile if I give you instructions?

burke · 2014-04-16T17:54:40Z

Yeah, I have a box I could run it on.

pnasrat · 2014-04-16T18:13:39Z

Also could you upload the compressed strace of the docker daemon whilst you
are doing rmi somewhere - preferably run with -f -ttt -T

On 16 April 2014 13:54, Burke Libbey notifications@github.com wrote:

Yeah, I have a box I could run it on.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3470#issuecomment-40630629
.

burke · 2014-04-16T19:06:42Z

https://gist.githubusercontent.com/burke/d0db19151ab28ef47c7b/raw/66a6550ff580b71612bc17faa4e078773632984c/rmi.log.gz

It only took about 10 seconds this time. It was closer to 2 minutes yesterday, but I recently pruned a few hundred dead containers and a few dozen older images.

burke · 2014-04-16T19:11:06Z

Actually in this case, it seems to be pretty reasonable. Mostly just unlinks. Damn. It was doing a LOT of graph stuff earlier.

crosbymichael · 2014-04-16T19:13:09Z

Sorry for the late repoen

vieux · 2014-05-15T22:47:26Z

can anybody reproduce with 0.11 ?

vincentwoo · 2014-06-07T19:28:30Z

I have not seen this problem post 0.11.

unclejack · 2014-06-18T12:21:09Z

@vincentwoo Thank you for the update. I'll close this now.

Please feel free to comment on this issue if you run into this problem.

ashneo76 · 2014-09-22T20:02:06Z

I am running in to the issue of docker hanging during docker rmi on Arch Linux still. And none of my images get removed.

~ % docker --version
Docker version 1.2.0, build fa7b24f

~ % docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f

unclejack · 2014-09-22T20:06:55Z

@ashneo76 Please provide the output of docker info, docker version and uname -a. Being given the output of those commands is the only information required to try to figure out what's going on.

ashneo76 · 2014-09-22T22:07:45Z

~ % docker info
^C%
~ % uname -a 
Linux flash 3.12.9-x86_64-linode37 #1 SMP Mon Feb 3 10:01:02 EST 2014 x86_64 GNU/Linux
# docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f

Funny, that docker info also hangs.

Looks like a linode bug. I tested on a physical machine (non-linode) with Arch linux and docker info worked fine.

# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-8:2-270345-pool
 Pool Blocksize: 64 Kb
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.16.1-1-ARCH
Operating System: Arch Linux
WARNING: No swap limit support
~ % uname -a
Linux flash2 3.16.1-1-ARCH #1 SMP PREEMPT Thu Aug 14 07:40:19 CEST 2014 x86_64 GNU/Linux
# docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f

unclejack · 2014-09-22T22:12:02Z

@ashneo76 Docker version is still missing.

ashneo76 · 2014-09-22T22:15:19Z

Updated the original post accordingly. Thanks. @unclejack

ichekrygin · 2015-04-16T18:52:46Z

docker version

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef

docker info

docker Containers: 2018
Images: 4273
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 8309
Execution Driver: native-0.2
Kernel Version: 3.13.0-48-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 15.56 GiB
Name: noname
ID: //
WARNING: No swap limit support

uname -a

Linux noname 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

time docker rmi 7eaa55ba2ffc

time docker rmi 7eaa55ba2ffc
Deleted: 7eaa55ba2ffc098945c53ff800fd135cbfb3dcd1386866a638aba8c3cb060b70
Deleted: 974d950694cc65e836fb64c54ec809467f7ef11d685ca755165ec86a1b3ee384
Deleted: fa6fbe71d7bac516b765b9e7faf7312c141c108671212592edd6b74229539858
Deleted: 47820837aeccb4be1f24a876ee797fdbb20ff4400307e884394be6ae3e751f99
Deleted: e9c9d64fc388ba19fc4301367a5fa4999b87cff34c4ffc39b9b3afcf63e0ca92
Deleted: 12e248d83a2b6ba93549942e09b2cd59ce693e2f4c73c9ff7b2f4a2536232f53
Deleted: 3f52f35c6f066f7ba0dfc6ffd1789e3ddc2d88aae82763a1b763a8e82d8ced1c
Deleted: b3e4083f210f2beeb0b83a14a3f1a6b62cc9b5e069e133056528f4ba0fa13721
0.02user 0.00system 1:26.05elapsed 0%CPU (0avgtext+0avgdata 6192maxresident)k
0inputs+0outputs (0major+1602minor)pagefaults 0swaps

top (while running rmi)

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                  
 7605 root      20   0  854412 113916   7520 R 102.6  0.7  12:19.10 docker

unclejack · 2015-04-17T13:45:09Z

@ichekrygin Can you update your system using the following command apt-get update && apt-get dist-upgrade -y && apt-get install lxc-docker and reboot, please?

Please comment on this issue if you're still seeing the problem afterwards.

ichekrygin · 2015-04-17T18:22:43Z

Thanks for the update. All I did was update lxc-docker to 1.6.0 and it seems to solve the issue (rmi goes much faster, although still > 100% cpu usage, but I guess that is expected).

mingfang · 2016-05-22T14:05:18Z

This problem is back in v1.11.1.
docker rmi docker images -q --filter dangling=truehangs docker to the point that evendocker info` does not return.
As a result, my kubernetes nodes stopped reporting status causing the entire cluster to fail.

thaJeztah · 2016-05-22T14:16:30Z

@mingfang possibly related to #22732, but be aware that (afaik) kubernetes is not yet compatible with docker 1.11

parvathy-g · 2016-05-27T17:55:50Z

This is happening to me for the following docker version
Client:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:25:01 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:25:01 UTC 2015
OS/Arch: linux/amd64

docker info
hangs

the following commands also hang.
docker rmi
docker rm
docker build

Please help me out .

thaJeztah · 2016-05-27T19:27:49Z

@parvathy-g are you using devicemapper (and on "loop" devices?), could be related to that

lancerushing · 2016-08-22T16:06:10Z

Same problem. But mine occurred after the VM running docker ran out of disk space. Now even $ docker info hangs.

Work Around by deleting my local docker files: $ sudo killall docker then $ sudo rm -rf /var/lib/docker and $ sudo service docker start

I assume one of the db files was corrupted with the machine ran out of disk space. (??)

sebastialonso · 2017-03-18T15:25:16Z

It's happening to me still. Only when the machine runs out of space, which is terrible, for I'm trying to delete images to gain space.

ttindell2 · 2017-06-12T22:42:53Z

I am having this same issue. Docker hangs on:

docker images
docker info

Here is the information:

docker version:

Client:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-28.git1398f24.el7.centos.x86_64
Go version: go1.7.4
Git commit: 1398f24/1.12.6
Built: Fri May 26 17:28:18 2017
OS/Arch: linux/amd64

Server:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-28.git1398f24.el7.centos.x86_64
Go version: go1.7.4
Git commit: 1398f24/1.12.6
Built: Fri May 26 17:28:18 2017
OS/Arch: linux/amd64

crosbymichael closed this as completed Jan 8, 2014

crosbymichael reopened this Apr 16, 2014

unclejack added the /project/point-o label Apr 30, 2014

vieux mentioned this issue May 15, 2014

"docker rmi" hangs #4429

Closed

vieux removed the /project/point-o label May 16, 2014

unclejack closed this as completed Jun 18, 2014

Docker RMI spins 100% CPU and takes a very long time to complete #3470

Docker RMI spins 100% CPU and takes a very long time to complete #3470

Comments

vincentwoo commented Jan 6, 2014

vincentwoo commented Jan 6, 2014

crosbymichael commented Jan 6, 2014

MichaelSp commented Jan 7, 2014

crosbymichael commented Jan 8, 2014

vincentwoo commented Jan 9, 2014

crosbymichael commented Jan 9, 2014

vincentwoo commented Jan 9, 2014

crosbymichael commented Jan 9, 2014

arthur-debert commented Jan 24, 2014

bmurphy1976 commented Feb 3, 2014

socketbind commented Feb 20, 2014

burke commented Apr 15, 2014

vieux commented Apr 16, 2014

burke commented Apr 16, 2014

pnasrat commented Apr 16, 2014

burke commented Apr 16, 2014

pnasrat commented Apr 16, 2014

burke commented Apr 16, 2014

pnasrat commented Apr 16, 2014

burke commented Apr 16, 2014

burke commented Apr 16, 2014

crosbymichael commented Apr 16, 2014

vieux commented May 15, 2014

vincentwoo commented Jun 7, 2014

unclejack commented Jun 18, 2014

ashneo76 commented Sep 22, 2014

unclejack commented Sep 22, 2014

ashneo76 commented Sep 22, 2014

unclejack commented Sep 22, 2014

ashneo76 commented Sep 22, 2014

ichekrygin commented Apr 16, 2015

unclejack commented Apr 17, 2015

ichekrygin commented Apr 17, 2015

mingfang commented May 22, 2016

thaJeztah commented May 22, 2016

parvathy-g commented May 27, 2016

thaJeztah commented May 27, 2016

lancerushing commented Aug 22, 2016

sebastialonso commented Mar 18, 2017

ttindell2 commented Jun 12, 2017