-
Notifications
You must be signed in to change notification settings - Fork 291
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"load metadata for" during docker pull hangs for about 10-20 seconds #10247
Comments
I agree that it looks like a DNS issue. Do you have slowness looking up the same DNS addresses from WSL 2 outside the context of Docker? |
@stephen-turner No, using
|
Hmmm, that's strange. We do have our own DNS server, but other people aren't reporting that it's slow. What's your environment like? Do you have any corporate security software that could be inserting itself on the network path? |
This is my personal machine. I'm not using any VPNs or anti virus software besides the default that Windows 10 Pro runs. This is actually a fresh install of Windows. Less than two weeks old. But I just tried disabling "Microsoft Defender Firewall" under "Private Network", and that improved it. For I don't understand right now why disabling the firewall helped. I'm also not very familiar with Windows security settings. I have more experience using Ubuntu for development. I remember using EDIT: Turns out it still hangs sometimes with docker pull, still for about 10 seconds, even with the firewall turned off. |
@stephen-turner Then I updated it to the latest version as of right now, which is So for me, the issue is affecting more than one computer and it only occurs when updating to 3.1.0, however I can't pinpoint exactly which version greater than 2.5.0.1 introduced the regression. |
I have the same issue, I recently reinstalled Windows twice and had this issue both times, but it wasn't present before that, so this might be a Windows/WSL 2 issue. |
Turns out this problem is introduced in Docker Desktop 3.1.0 (which takes 59 seconds for me to pull If there is any older version than 3.1.0 and newer than 3.0.4, I would love to try it. Until this gets fixed, I'm staying on 3.0.4. |
Nice to see another person confirm the issue so I know I wasn't crazy. I've just been putting up with the delay since upgrading to 3.1.0 (and since reformatting my other machines, which I do quite frequently). But it sure would be nice to have instant pulls again. |
I don't really know what to make of this. Given that we only have two people reporting it, I feel that it must be something very specific in your environment that is interacting with a change in Docker Desktop, but I'm not sure what. |
True, it interrupts flow a lot, so I feel like if it were affecting more than 2 people, we'd see more reports of it here. I can try tweaking my environment on my computers to see if I can get it to stop. So far it's affected every computer I've used Docker Desktop and WSL2 on through multiple reformats. |
Turns out it was actually a DNS issue - with IPv6. I disabled IPv6 via adapter settings in Windows on the main adapter (Ethernet for me) - this worked flawlessly. Then I set up CloudFlare's 1.1.1.1 DNSv6 and set a static IPv6. This actually worked and as far as I can see, times are the same as before on 3.0.1. If it helps debugging this issue, I have 2.5GbE Realtek PCIe network card. |
Interesting. I'll look into DNS issues on my end to see if that's the culprit too. |
possibly caused by this? microsoft/WSL#4901 |
@chuanqisun No, the issue didn't come up for me until that particular new version of Docker Desktop for Windows. Also, my internet connection is very fast inside WSL (I get my full 1Gbit/s up and down), it's just there is always a very long pause the first time a request for a particular Docker image is made. Other requests like doing an The issue continues to affect me for all computers I use Docker Desktop on, throughout each update there has been since this was reported. |
Reporting in, been baffled by this exact bug for the last few days. No matter what image I pull it always takes 10 seconds to load the metadata. :\ Tried a different DNS as well. |
@Kleptine Can't say I'm happy to hear you're experiencing this, because I know how frustrating it is. But I am glad to see a 3rd person reporting in also experiencing it. Hopefully with more reports, we can get it on the maintainers' radar. |
I noticed something interesting today. tl;dr logging into Docker Hub removed this delay. But you have to log in before each pull step. Details: I googled the error tonight and found docker/buildx#476 where one of the error messages someone posts includes the substring After logging in, I pulled a few more images. To reproduce my problem, I have to pull an image I've never tried to pull before. I worked my way down Node.js versions until I got to version 6, which was a version I'd never pulled before. Presumably, Docker would have to fetch metadata for this version. It pulled instantly:
BUT this only works if I log in again every time I want to pull an image I've never pulled before. If I log in, pull one (which will be instant), and then try to pull another, I get the delay. But if I write a one liner like Perhaps this delay is caused by some sort of issue with tokens, where tokens can only be used once and are refreshed after a 10 second timeout? Unfortunately, this workaround doesn't solve most of the pain this problem causes because most of the pain comes from the many pull steps that occur in processes like building an image or starting a docker-compose stack. I can't have it run EDIT: I tested this with another one liner ( |
I tried a After logging in, the subsequent However, the immediate build after, it's back to 10 seconds again. This doesn't actually seem to work-around the problem, just moves it to the For reference, I am also using WSL2. I installed docker desktop on my personal computer just a few days back, so everything is a fresh install here. Nothing fancy on my machine, it's just my personal home box. |
Yeah it's some sort of networking issue. I confirmed earlier that it happened when I upgraded from Docker Desktop |
Just wanted to report that I am experiencing this too. I am on a fairly fresh Windows 10 machine running WSL 2 and docker 3.3.3. Running
I also observed that if I run a build multiple times in quick succession (without login), the first one is slow and the next ones are fast. After waiting about 10-20 seconds and the build becomes slow again. So it appears that any communication with docker.io is initially slow, then its fast as long as you are running commands quickly, and then becomes slow again after a ~10s pause. |
If you're reading this and you're experiencing it, speak up! We need as many comments as possible left on this issue so that it can get attention from the devs! I will say, having that mandatory pause while my images pull has done wonders for my mental health. It's nice to take breaks now and then, and remember to smell the roses. |
We managed to work around this issue (on Debian, Docker Version 20.10.7) by explicitly -not- using Docker BuildKit: To see whether Docker BuildKit is enabled: Unset: Worked without hanging afterwards. |
I can reproduce this 100%, same issue |
I can confirm that I don't have DOCKER_BUILDKIT enabled, and still see the issue. |
@kakuzei thanks for the update. With 4.17.0, could you verify exactly which
where Assuming that looks ok, could you try running
to see if the latency from running FYI I did a few more experiments with my
I mention this because it's quite confusing behaviour when testing. |
@djs55 thank you for your answer. Here are the paths of the docker-credential commands:
Please find here the output of some commands when
The latest command executed shows the issue. |
@kakuzei thanks for the detailed update. On my machine I notice that
Could you try deleting |
@djs55 Many thanks for that: it solves the 5 minutes delay issue. Could you confirm me the buggy |
it seems v4.17.1 fixed the delay for me |
👍 I also did the following after uninstalling v4.16.2, before installing v4.17.1. Just in case you still see the problem with 4.17.1, you could try this out.
|
There hasn't been any activity on this issue for a long time. Prevent issues from auto-closing with a /lifecycle stale |
no |
I'm facing the same problem. It always takes 31-32 seconds to "load metadata". When I enabled debug logging for the docker daemon, I noticed that it got stuck while requesting one of my registry-mirrors (not the first one, though). I tried using curl to test the mirror site and found out that it was actually down. After removing that mirror, the build process became super fast again. |
Yeah still an issue for me |
same, still an issue. |
This same: Docker Desktop 4.24.1 (Docker version 24.0.6, build ed223bc) About 80 seconds for |
I'm experiencing this issue on my Macbook Pro M2.
Relevant: I'm on a Wi-Fi hotspot at the airport, sharing from my own phone. |
I had a similar issue that I boiled down to this command.
You'll notice the When specifying the quad-8 DNS server in the
The equivalent behavior can be achieved by setting the DNS in the
My setup: Windows 11, Docker Desktop 4.27.0-135262, Engine 25.0.1 |
Following the advice in https://stackoverflow.com/a/70483395/1449366 and just pulled the problematic images manually. |
Does anybody know what triggers |
BTW I was not able to solve this problem by removing
|
I checked the steps described in https://docs.docker.com/docker-for-windows/troubleshoot/ and I did the "purge data" and "reset to factory settings" options. Neither fixed my issue.
I used to have none of these pauses using Docker on WSL2. I'd run "docker pull" or "docker build" and it would begin immediately. I'm not sure if it's the latest update for me (this is a fresh Windows 10 install) that's causing this, but now it's very slow when I start. When I run
docker pull postgres
it hangs onUsing default tag: latest
for about 10 seconds. It also hangs this way when I rundocker pull postgre
(a non-existent image) it still has the pause. So I'm thinking it might be related to DNS.When I tried doing a build I saw this happen again, because now my build output is different. It shows more steps and has color. The very first step mentioned getting metadata, and that step always took 10 seconds:
After the step finishes, the rest proceeds quickly.
Since I'm caching Python dependencies, the rest finishes instantly since it just has to copy in my changed source code files, but I still have to wait that 10 seconds each time:
My Dockerfile, for reference:
Note that this affects all images I tried, not just
postgres
andpython
. I also found that the pause withdocker pull
happened withnode
,mongo
,openjdk
, andamazon/opendistro-for-elasticsearch
.Actual behavior
docker pull
operations take a long time to start.docker build
operations take 10 seconds to finish the first step which appears to be doing nothing.Expected behavior
docker pull
begins downloading the first layer of the image quickly anddocker build
's first step finishes almost instantly instead of 10 seconds.Information
Please, help us understand the problem. For instance:
Steps to reproduce the behavior
Described above.
The text was updated successfully, but these errors were encountered: