docker pull: always show me "Layer already being pulled by another client. Waiting." #15603
Comments
|
This looks very similar to what #15646 tries to solve. Only thing I don't understand is that how this could appear on the first request after restart as we don't store the pulling pool on the disk. That VM was rebooted after Mac restart? Not restored from snapshot/state? |
|
I ran into this problem as well and resolved it by stopping the VM, removing all my unused images, and then restarting the VM. Not sure if removing the images was necessary, but I'd wanted to clean them up anyway. |
|
Hi tonistiigi, But this morning after my Mac upgraded and restarted, I've pulled the redis image successfully. |
|
USER POLL The best way to get notified when there are changes in this discussion is by clicking the Subscribe button in the top right. The people listed below have appreciated your meaningfull discussion with a random +1: |
|
+1 (Just testing the turtle!) |
|
Have only one Docker Machine and freshly booted. Tried to run the containers using the https://github.com/javaee-samples/docker-java/blob/master/attendees/cicd/docker-compose.yml and stuck at the message "Layer already being pulled by another client. Waiting.". Restarted the Docker Machine. Restarted Docker Machine again, and finally got it working. Here are version details: |
|
Building an image that inherits from Dockerfile is at: https://github.com/javaee-samples/docker-java/blob/master/attendees/cicd/jenkins/Dockerfile. Just change |
|
get the issue also , I try to restart docker and delete images even server reboot,the issue still exits ^Croot@Ubuntu-14-x8664:~# docker version Server: |
|
Had the same issue pulling mysql. root@drop:~# docker version Server: Killed the docker process, removed all the images with |
|
so do I! |
|
Same issue here for |
|
Seeing this issue as well attempting to pull postgres:latest in a docker within a docker image |
|
I've been evaluating Docker for our shop the past couple of days. I must say it's disheartening when the very first thing I try to do requires 13 restarts of the Docker daemon to complete successfully. Windows + boot2docker VM and also LinuxMint (17.2): |
|
It's unfortunate the patch which probably fixes this didn't make it into 1.8.2 |
|
This issue became a head pain for me. I located in Ukraine. And two days ago I had this issue only on few images. Now I can't retrieve any images at all. I think it's connected with Amazon Cloudfront somehow. I have Digital Ocean droplet in the US. So I installed tinyproxy there and started docker daemon manually with HTTP_PROXY env variable. And bang, image pulled successfully. |
|
On my slow/unreliable home connection I've had problems downloading images since the pre-1.0 days and still to this day. Large/complex images are hit or miss, and then I often run into this issue. What I do is pull the image on a remote server, Is there a way to serialize the download of layers? So it would start at the first layer I don't have and then continue from there, one at a time? That would probably help. |
|
Happened very frequently for me on both OS X (with Docker Toolbox 1.8.1). It was definitely more apparent on a slower connection (needed to reboot the VM roughly 5 times for a relatively small image), but it still happened pretty frequently on a 100mbit connection. Since upgrading to 1.8.2 it seems to be less frequent (also dumped all of my images and containers). I was able to download ubuntu:14.04 and tutum/wordpress without rebooting the VM once. For reference yesterday I had to restart the VM about 20 times just to get the Wordpress image. It still happens occasionally on 1.8.2 on my Macbook (OS X 10.9.5), but it usually only happens on very large images now on a slower connection. In terms of upgrading from OS X I believe you have to download the whole Toolbox again and just rerun the installer, then when it's done (I believe) you have to create a new image to update the Server version, so maybe give that a try. I believe you will lose your images in the transition using this method. You could also try Version Info |
|
I started the docker daemon with debug-logging enabled so I could see what happend directly before the message "Layer already being pulled by another client. Waiting." appears. It seems docker tries to restart the download off all layers (even those that are already pulled) on any error without cleaning up the previous download. For me this happend on two occassions:
In both cases the message appears, and docker deadlocks itself. |
|
I am expiriencing same issue on linux with version 1.8.2: this is a sample of what happens: What's the solution?! |
|
I'm also having frequent troubles with pulling images today. Everything works like a charm. See issue:
docker.log: |
|
@elyulka excessive inode usage is a known issue with overlay fs. |
|
I am also facing the same issue while downloading any docker images its taking long hours and image is not getting downloaded... |
|
I get this issue, even after |
|
@tinco The reason why you get that message right away is that this image contains a duplicate layer. This should be unrelated to this issue and not cause blocking on pulls. The issue that caused creating such layers on pushes should be fixed with #14421, that is included in For the issue that causes blocking, its caused by |
|
+1 |
|
+1 |
|
+1 |
|
for all the people +1'ing with client and server version can you follow up and say what resolved your issue or if it was just waiting X amount of time and trying again? Did you have to force remove containers or reboot your VM. Information like that is much more helpful as we can see that the issue is indeed occurring on the current client. If you want to subscribe use the button on the right. Potentially times at which you are having in an issue is also helpful. |
|
I haven't solved the issue, but what I do is to restart docker service and |
|
A workaround that works for me is to restart the docker daemon. |
|
I haven't solved the issue either and even when I restart docker service and/or delete the VM, I have to try more than once in a seemingly random pattern until it eventually works. |
|
Can confirm this @mallegrini , sometimes it is solved by restarting the docker daemon and/or the VM. But it does not seem to be reproducible at all times |
|
Let me restate it: what I mean is that I need to try more than once in order to pull the image successfully. Sometimes (very very rarely) it works on first try. Most of the times I have to restart docker, pull again (and fail), restart, pull, etc until it finally succeeds. I'm trying right now, but this is weird: on my Mac I have also a Linux VM with docker 1.6.2 that used to pull images flawlessy 3 days ago while 1.8.2 couldn't. Today I'm getting timeout error even pulling images with docker 1.6.2 Get https://index.docker.io/v1/repositories/library/wordpress/images: dial tcp 54.210.246.75:443: connection timed outI ruled out server/network problems because docker 1.6.2 used to work but now I'm clueless... |
|
You may laugh, but I have been looking at this myself and at a whim started docker using strace: Then I ran the client trying to pull an image that I knew was consistently failing unless the docker daemon was restarted multiple times (up to 20 times until it worked). To my surprise, albeit slow, the download finished successfully. So maybe this slowdown by strace is helping the daemon to not run into a weird race condition between download/verification processes which cause the verification of an image to fail for no apparent reason. I know this is a wild guess but figured I'd add it to this discussion. EDIT: 4/4 successful EDIT 2: It seems that this happens when two images are running Checking the debug log of the running daemon, I can confirm these two have failed the checksum check in the logs: This in turn causes the system to fallback to v1 protocol which then errors out with an |
|
For me, this issue only happens when I'm impatient and |
|
I get this issue with |
|
@rodolfo42 I should have clarified, this is even before the issue with aborting a download. Checksum validation simply fails, maybe I should have opened or checked for another ticket? |
|
@icecrime Is there an associated version of |
|
FWIW, I installed docker 1.9 and pulled |
|
Thanks for testing @dmitrym0 ! |
|
Works for me too. |
|
👍 |
|
thanks, can't wait to use it |
|
Just a note, which may help others. In my case, the error with the message 'Layer already being pulled by another client. Waiting.' appears to have been caused by using the 'boot2docker' command instead of docker-machine (or perhaps intermixing the two). I removed /usr/local/bin/boot2docker, deleted ~/.docker, deleted all docker-related VMs, reinstalled from the "Docker Toolkit", built the 'default' docker-machine, and have not had this problem since. Docker Version (client/server) 1.8.3 (build from commit f4bf5c7); docker-machine version 0.4.1 (e2c88d6) . |
|
@Terry-Weymouth likely a coincidence. I have only ever had docker-machine installed on my computer and I have run into this issue 4 or 5 times. |
|
I have been having this issue with docker-machine. A simple stop and start seems to resolve the issue for now. |
|
@powdahound I got an error trying to xargs docker rmi after the machine was stopped. This worked though, thanks for the tip! $ docker-machine stop default |
|
Happens quite often on |
and others
Question
Yesterday I tried the docker 1.8 on my Mac pro, and created two docker machines (default and dev) first and then removed the dev machine.
Today when I tried to pull the redis image but always got the "Layer already being pulled by another client. Waiting." error message even I reboot my mac. What should I do to solve it?
Environment
➜ ~ docker version
Client:
Version: 1.8.0
API version: 1.20
Go version: go1.4.2
Git commit: 0d03096
Built: Tue Aug 11 17:17:40 UTC 2015
OS/Arch: darwin/amd64
Server:
Version: 1.8.0
API version: 1.20
Go version: go1.4.2
Git commit: 0d03096
Built: Tue Aug 11 17:17:40 UTC 2015
OS/Arch: linux/amd64
➜ ~ docker info
Containers: 2
Images: 41
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 45
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.0 (TCL 6.3); master : 7f12e95 - Tue Aug 11 17:55:16 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: default
ID: FAS3:MCST:CB2E:IFIE:KTB6:EIDL:YGST:6I65:N64P:AQ6O:5J7N:OHJN
Debug mode (server): true
File Descriptors: 15
Goroutines: 25
System Time: 2015-08-15T02:03:44.917069573Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Username: easonyi
Registry: https://index.docker.io/v1/
Labels:
provider=virtualbox
➜ ~ uname -a
Darwin Eason 14.4.0 Darwin Kernel Version 14.4.0: Thu May 28 11:35:04 PDT 2015; root:xnu-2782.30.5~1/RELEASE_X86_64 x86_64
Step 1:
Last login: Fri Aug 14 23:35:21 on ttys006
------Welcome back!------ [2015-08-15 06:35:26]
➜ ~ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
default virtualbox Stopped
➜ ~ docker-machine start default
Starting VM...
Started machines may have new IP addresses. You may need to re-run the
docker-machine envcommand.➜ ~ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
default * virtualbox Running tcp://192.168.99.100:2376
➜ ~ eval $(docker-machine env default)
➜ ~ docker pull redis
Using default tag: latest
latest: Pulling from library/redis
a81e536d75d2: Pull complete
b5385b7b49ae: Pull complete
5eced7370894: Pull complete
9b5f6b31f2b8: Pull complete
68145731e6c3: Pull complete
e07b98fdf391: Pull complete
e53cade0786f: Pull complete
d41c1101f46b: Pulling fs layer
76ef21f24f06: Download complete
4736cd3389a7: Download complete
d794940dc881: Download complete
bfb2f87f9ad0: Download complete
1c03c2a5aa29: Pulling fs layer
2ff62b1c4295: Download complete
0ff407d5a7d9: Download complete
0ff407d5a7d9: Layer already being pulled by another client. Waiting.
60c52dbe9d91: Already exists
Pulling repository docker.io/library/redis
^C%
➜ ~ docker pull redis
Using default tag: latest
latest: Pulling from library/redis
d41c1101f46b: Pulling fs layer
76ef21f24f06: Pulling fs layer
d41c1101f46b: Verifying Checksum
d794940dc881: Layer already being pulled by another client. Waiting.
bfb2f87f9ad0: Layer already being pulled by another client. Waiting.
1c03c2a5aa29: Layer already being pulled by another client. Waiting.
2ff62b1c4295: Layer already being pulled by another client. Waiting.
0ff407d5a7d9: Layer already being pulled by another client. Waiting.
0ff407d5a7d9: Layer already being pulled by another client. Waiting.
60c52dbe9d91: Already exists
a81e536d75d2: Already exists
b5385b7b49ae: Already exists
5eced7370894: Already exists
9b5f6b31f2b8: Already exists
68145731e6c3: Already exists
e07b98fdf391: Already exists
e53cade0786f: Already exists
Pulling repository docker.io/library/redis
➜ ~ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
mysql latest c45e4ba02f47 2 days ago 283.8 MB
nginx latest 6886fb5a9b8d 3 weeks ago 132.9 MB
e53cade0786f 3 weeks ago 100.1 MB
Step 2:
After I rebooted my mac pro, I've tried again and got the same error:
Last login: Sat Aug 15 07:58:54 on ttys000
------Welcome back!------ [2015-08-15 07:59:29]
➜ ~ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
default virtualbox Stopped
➜ ~ docker-machine start default
Starting VM...
Started machines may have new IP addresses. You may need to re-run the
docker-machine envcommand.➜ ~ eval $(docker-machine env default)
➜ ~ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
default * virtualbox Running tcp://192.168.99.100:2376
➜ ~ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
mysql latest c45e4ba02f47 2 days ago 283.8 MB
nginx latest 6886fb5a9b8d 3 weeks ago 132.9 MB
e53cade0786f 3 weeks ago 100.1 MB
➜ ~ docker pull redis
Using default tag: latest
latest: Pulling from library/redis
d41c1101f46b: Pull complete
76ef21f24f06: Pull complete
4736cd3389a7: Pull complete
d794940dc881: Pull complete
bfb2f87f9ad0: Pull complete
1c03c2a5aa29: Pulling fs layer
2ff62b1c4295: Download complete
0ff407d5a7d9: Download complete
4c8cbfd2973e: Already exists
60c52dbe9d91: Already exists
a81e536d75d2: Already exists
b5385b7b49ae: Already exists
5eced7370894: Already exists
9b5f6b31f2b8: Already exists
68145731e6c3: Already exists
e07b98fdf391: Already exists
e53cade0786f: Already exists
Pulling repository docker.io/library/redis
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/redis/images. You may want to check your internet connection or if you are behind a proxy.
➜ ~ docker pull redis
Using default tag: latest
latest: Pulling from library/redis
1c03c2a5aa29: Pull complete
2ff62b1c4295: Layer already being pulled by another client. Waiting.
0ff407d5a7d9: Layer already being pulled by another client. Waiting.
4c8cbfd2973e: Already exists
60c52dbe9d91: Already exists
a81e536d75d2: Already exists
b5385b7b49ae: Already exists
5eced7370894: Already exists
9b5f6b31f2b8: Already exists
68145731e6c3: Already exists
e07b98fdf391: Already exists
e53cade0786f: Already exists
d41c1101f46b: Already exists
76ef21f24f06: Already exists
4736cd3389a7: Already exists
d794940dc881: Already exists
bfb2f87f9ad0: Already exists
^C%
➜ ~
The text was updated successfully, but these errors were encountered: