Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"load metadata for" during docker pull hangs for about 10-20 seconds #10247

Open
4 of 5 tasks
mattwelke opened this issue Jan 29, 2021 · 223 comments
Open
4 of 5 tasks

"load metadata for" during docker pull hangs for about 10-20 seconds #10247

mattwelke opened this issue Jan 29, 2021 · 223 comments

Comments

@mattwelke
Copy link

mattwelke commented Jan 29, 2021

I checked the steps described in https://docs.docker.com/docker-for-windows/troubleshoot/ and I did the "purge data" and "reset to factory settings" options. Neither fixed my issue.

I used to have none of these pauses using Docker on WSL2. I'd run "docker pull" or "docker build" and it would begin immediately. I'm not sure if it's the latest update for me (this is a fresh Windows 10 install) that's causing this, but now it's very slow when I start. When I run docker pull postgres it hangs on Using default tag: latest for about 10 seconds. It also hangs this way when I run docker pull postgre (a non-existent image) it still has the pause. So I'm thinking it might be related to DNS.

When I tried doing a build I saw this happen again, because now my build output is different. It shows more steps and has color. The very first step mentioned getting metadata, and that step always took 10 seconds:

image

After the step finishes, the rest proceeds quickly.

Since I'm caching Python dependencies, the rest finishes instantly since it just has to copy in my changed source code files, but I still have to wait that 10 seconds each time:

image

My Dockerfile, for reference:

FROM python:3.9-buster

ENV PYTHONUNBUFFERED=1 \
  POETRY_HOME=/opt/poetry \
  POETRY_NO_INTERACTION=1 \
  POETRY_VERSION=1.1.4 \
  POETRY_VIRTUALENVS_CREATE=false

ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"

WORKDIR /usr/src/app

RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

COPY pyproject.toml poetry.lock /usr/src/app/

RUN poetry install --no-dev

COPY wisdom_demo_beacon_generator wisdom_demo_beacon_generator
COPY searches_top_100.csv searches_top_100.csv

RUN poetry install --no-dev

CMD [ "python", "wisdom_demo_beacon_generator/main.py" ]

Note that this affects all images I tried, not just postgres and python. I also found that the pause with docker pull happened with node, mongo, openjdk, and amazon/opendistro-for-elasticsearch.

  • I have tried with the latest version of Docker Desktop
  • I have tried disabling enabled experimental features
  • I have uploaded Diagnostics
  • Diagnostics ID: 8001D9A7-6230-4B22-B6D3-BD9DB6909FB9/20210129041213
  • I am using the Docker WSL2 Backend
  • I am using the Docker Hyper-V Backend

Actual behavior

docker pull operations take a long time to start. docker build operations take 10 seconds to finish the first step which appears to be doing nothing.

Expected behavior

docker pull begins downloading the first layer of the image quickly and docker build's first step finishes almost instantly instead of 10 seconds.

Information

Please, help us understand the problem. For instance:

  • Is it reproducible? Yes. You can purge data and reset to factory settings, and then run the operations mentioned from a WSL2 distro like Ubuntu.
  • Is the problem new? Yes.
  • Did the problem appear with an update? Unsure. I reformatted Windows after changing machine's SSD to a new one. Then I installed Docker Desktop. It would have downloaded the latest version as I installed it.
  • A reproducible case if this is a bug, Dockerfiles with reproduction inside is best.
  • Windows Version: Windows 10 Pro 20H2 19042.746
  • Docker Desktop Version: 3.1.0 (51484)
  • WSL2 or Hyper-V backend? WSL2
  • Are you running inside a virtualized Windows e.g. on a cloud server or on a mac VM: No

Steps to reproduce the behavior

Described above.

@stephen-turner
Copy link
Contributor

I agree that it looks like a DNS issue. Do you have slowness looking up the same DNS addresses from WSL 2 outside the context of Docker?

@mattwelke
Copy link
Author

mattwelke commented Jan 29, 2021

@stephen-turner No, using dig from WSL 2 is instant, including for popular domains like google.com and for one of the domains the docker build commands used:

~ > dig docker.io

; <<>> DiG 9.16.1-Ubuntu <<>> docker.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44701
;; flags: qr rd ad; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;docker.io.                     IN      A

;; ANSWER SECTION:
docker.io.              0       IN      A       52.55.43.248
docker.io.              0       IN      A       3.220.75.233
docker.io.              0       IN      A       34.192.114.9
docker.io.              0       IN      A       34.204.125.5
docker.io.              0       IN      A       52.6.170.51
docker.io.              0       IN      A       34.200.7.11

;; Query time: 10 msec
;; SERVER: 172.26.160.1#53(172.26.160.1)
;; WHEN: Fri Jan 29 12:26:09 EST 2021
;; MSG SIZE  rcvd: 132

@stephen-turner
Copy link
Contributor

Hmmm, that's strange. We do have our own DNS server, but other people aren't reporting that it's slow.

What's your environment like? Do you have any corporate security software that could be inserting itself on the network path?

@mattwelke
Copy link
Author

mattwelke commented Jan 29, 2021

This is my personal machine. I'm not using any VPNs or anti virus software besides the default that Windows 10 Pro runs. This is actually a fresh install of Windows. Less than two weeks old.

But I just tried disabling "Microsoft Defender Firewall" under "Private Network", and that improved it. For docker pull, it's instant every time now. For docker build, there's still a 10 second hang each time the image is built with new content in my source code files that hasn't been built before.

I don't understand right now why disabling the firewall helped. I'm also not very familiar with Windows security settings. I have more experience using Ubuntu for development. I remember using ufw in Ubuntu to let apps and ports through, so I assume Windows has controls for this too. I could look into tweaking them, but I'd also need to know what apps/ports to allowlist in the settings, and why Docker in WSL 2 requires them to be allowlisted when it didn't before.

EDIT:

Turns out it still hangs sometimes with docker pull, still for about 10 seconds, even with the firewall turned off.

@mattwelke
Copy link
Author

@stephen-turner
I also have another computer that I hadn't updated Docker Desktop for in a while. I was running 2.5.0.1 (49550) also on WSL 2 also using Ubuntu 20.04, before I updated Docker Desktop it a few minutes ago. I tested out some docker pulls, and they were all instant. No problems, the same experience I've always had with both of my computers.

Then I updated it to the latest version as of right now, which is 3.1.0, and it now also has the problem where the docker pull operations hangs for about 10 seconds.

So for me, the issue is affecting more than one computer and it only occurs when updating to 3.1.0, however I can't pinpoint exactly which version greater than 2.5.0.1 introduced the regression.

@Timic3
Copy link

Timic3 commented Mar 14, 2021

I have the same issue, docker pull hangs for exactly 30 seconds on WSL 2 - I tried using VM with Debian and it was instant. Disabling firewall like you mentioned actually helped and made pulling instant, although I'm not sure I want to have firewall disabled. Any decent workaround for this?

I recently reinstalled Windows twice and had this issue both times, but it wasn't present before that, so this might be a Windows/WSL 2 issue.

@Timic3
Copy link

Timic3 commented Mar 14, 2021

Turns out this problem is introduced in Docker Desktop 3.1.0 (which takes 59 seconds for me to pull docker/getting-started image), while Docker Desktop 3.0.4 pulls almost instantaneously (4 seconds).

If there is any older version than 3.1.0 and newer than 3.0.4, I would love to try it. Until this gets fixed, I'm staying on 3.0.4.

@mattwelke
Copy link
Author

Nice to see another person confirm the issue so I know I wasn't crazy. I've just been putting up with the delay since upgrading to 3.1.0 (and since reformatting my other machines, which I do quite frequently). But it sure would be nice to have instant pulls again.

@stephen-turner
Copy link
Contributor

I don't really know what to make of this. Given that we only have two people reporting it, I feel that it must be something very specific in your environment that is interacting with a change in Docker Desktop, but I'm not sure what.

@mattwelke
Copy link
Author

True, it interrupts flow a lot, so I feel like if it were affecting more than 2 people, we'd see more reports of it here. I can try tweaking my environment on my computers to see if I can get it to stop. So far it's affected every computer I've used Docker Desktop and WSL2 on through multiple reformats.

@Timic3
Copy link

Timic3 commented Mar 16, 2021

Turns out it was actually a DNS issue - with IPv6. I disabled IPv6 via adapter settings in Windows on the main adapter (Ethernet for me) - this worked flawlessly. Then I set up CloudFlare's 1.1.1.1 DNSv6 and set a static IPv6. This actually worked and as far as I can see, times are the same as before on 3.0.1.

If it helps debugging this issue, I have 2.5GbE Realtek PCIe network card.

@mattwelke
Copy link
Author

mattwelke commented Mar 17, 2021

Interesting. I'll look into DNS issues on my end to see if that's the culprit too.

@chuanqisun
Copy link

possibly caused by this? microsoft/WSL#4901

@mattwelke
Copy link
Author

@chuanqisun No, the issue didn't come up for me until that particular new version of Docker Desktop for Windows. Also, my internet connection is very fast inside WSL (I get my full 1Gbit/s up and down), it's just there is always a very long pause the first time a request for a particular Docker image is made. Other requests like doing an apt-get update work instantly.

The issue continues to affect me for all computers I use Docker Desktop on, throughout each update there has been since this was reported.

@Kleptine
Copy link

Reporting in, been baffled by this exact bug for the last few days. No matter what image I pull it always takes 10 seconds to load the metadata. :\ Tried a different DNS as well.

@mattwelke
Copy link
Author

@Kleptine Can't say I'm happy to hear you're experiencing this, because I know how frustrating it is. But I am glad to see a 3rd person reporting in also experiencing it. Hopefully with more reports, we can get it on the maintainers' radar.

@mattwelke
Copy link
Author

mattwelke commented May 10, 2021

I noticed something interesting today. tl;dr logging into Docker Hub removed this delay. But you have to log in before each pull step.

Details:

I googled the error tonight and found docker/buildx#476 where one of the error messages someone posts includes the substring error getting credentials. They must have had some sort of debug output mode on to see this. Or, maybe it only shows this if there's an error (and we're only experiencing a delay followed by eventual success).

After logging in, I pulled a few more images. To reproduce my problem, I have to pull an image I've never tried to pull before. I worked my way down Node.js versions until I got to version 6, which was a version I'd never pulled before. Presumably, Docker would have to fetch metadata for this version. It pulled instantly:

~ > docker login
Authenticating with existing credentials...
Login Succeeded
~ > docker pull node:8
8: Pulling from library/node
146bd6a88618: Already exists
9935d0c62ace: Already exists
db0efb86e806: Already exists
e705a4c4fd31: Already exists
c877b722db6f: Already exists
645c20ec8214: Already exists
db8fbd9db2fe: Already exists
1c151cd1b3ea: Already exists
fbd993995f40: Already exists
Digest: sha256:a681bf74805b80d03eb21a6c0ef168a976108a287a74167ab593fc953aac34df
Status: Downloaded newer image for node:8
docker.io/library/node:8
~ > docker pull node:6
6: Pulling from library/node
c5e155d5a1d1: Pull complete
221d80d00ae9: Pull complete
4250b3117dca: Pull complete
3b7ca19181b2: Pull complete
425d7b2a5bcc: Pull complete
69df12c70287: Pull complete
ea2f5386a42d: Pull complete
d421d2b3c5eb: Pull complete
Digest: sha256:e133e66ec3bfc98da0440e552f452e5cdf6413319d27a2db3b01ac4b319759b3
Status: Downloaded newer image for node:6
docker.io/library/node:6

BUT this only works if I log in again every time I want to pull an image I've never pulled before. If I log in, pull one (which will be instant), and then try to pull another, I get the delay. But if I write a one liner like docker login; docker pull python:3.4; docker login; docker pull python:3.3; docker login; docker pull python:3.2, each will pull instantly.

Perhaps this delay is caused by some sort of issue with tokens, where tokens can only be used once and are refreshed after a 10 second timeout?

Unfortunately, this workaround doesn't solve most of the pain this problem causes because most of the pain comes from the many pull steps that occur in processes like building an image or starting a docker-compose stack. I can't have it run docker login before each pull step in such processes.

EDIT:

I tested this with another one liner (for VERSION in 10 9.6 9.5 9.4; do docker login; docker pull postgres:$VERSION; done) and noticed that the first login took 10 seconds too. Then, each login after that was instant. All pulls were instant.

@Kleptine
Copy link

Kleptine commented May 10, 2021

I tried a docker login and that command also took about 10-15 seconds! Brutal.

After logging in, the subsequent docker build metadata step is only 1 second.

However, the immediate build after, it's back to 10 seconds again. This doesn't actually seem to work-around the problem, just moves it to the docker login command. But it does seem to show this is some sort of issue with dockerhub and windows networking.

For reference, I am also using WSL2. I installed docker desktop on my personal computer just a few days back, so everything is a fresh install here. Nothing fancy on my machine, it's just my personal home box.

@Kleptine
Copy link

Kleptine commented May 10, 2021

I tried disabling IPv6 on the network adapter, which results in the following behavior:

On the first run after, I get a new line, which shows up while it is loading metadata:
=> [auth] library/ubuntu:pull token for registry-1.docker.io
image

This first run still takes 10 seconds. On the next few subsequent runs, the metadata load is instant, back to where it should be. However, after waiting for a little while (a few minutes in my tests). The next docker build will again take 10 seconds on metadata.

So IPv6 improves things but does not solve the issue, and disabling IPv6 system wide is not really a useable workaround. The fact that I got new output definitely makes me think it's a networking issue for sure.

@mattwelke
Copy link
Author

Yeah it's some sort of networking issue. I confirmed earlier that it happened when I upgraded from Docker Desktop 2.5.0.1 to 3.1.0. Another user commented that for them, it happened when they updated from 3.0.4 to 3.1.0. So it looks like 3.1.0 is the update that introduced the regression.

@DesignByOnyx
Copy link

DesignByOnyx commented Jun 11, 2021

Just wanted to report that I am experiencing this too. I am on a fairly fresh Windows 10 machine running WSL 2 and docker 3.3.3.

Running docker login seemed to momentarily fix the issue with the following observation:

  • It took roughly 10-12 seconds to login
  • I was able to run docker build 20 times in quick succession without the lag (up + Enter, up + Enter, ...)
  • After waiting about 10 - 20 seconds docker build was slow again

I also observed that if I run a build multiple times in quick succession (without login), the first one is slow and the next ones are fast. After waiting about 10-20 seconds and the build becomes slow again. So it appears that any communication with docker.io is initially slow, then its fast as long as you are running commands quickly, and then becomes slow again after a ~10s pause.

@mattwelke
Copy link
Author

If you're reading this and you're experiencing it, speak up! We need as many comments as possible left on this issue so that it can get attention from the devs! I will say, having that mandatory pause while my images pull has done wonders for my mental health. It's nice to take breaks now and then, and remember to smell the roses.

image

@mattwelke mattwelke changed the title "docker pull" hangs on "Using default tag:" and "docker build" hangs on "=> [internal] load metadata for docker.io/library/<img>" "docker pull" hangs for about 10 seconds except immediately after "docker login" Jun 15, 2021
@kaner
Copy link

kaner commented Jul 8, 2021

We managed to work around this issue (on Debian, Docker Version 20.10.7) by explicitly -not- using Docker BuildKit:

To see whether Docker BuildKit is enabled:
echo $DOCKER_BUILDKIT

Unset:
unset DOCKER_BUILDKIT
Or:
export DOCKER_BUILDKIT=

Worked without hanging afterwards.

@mattwelke
Copy link
Author

Didn't work for me. Looks like I wasn't using BuildKit in the first place. If I run echo $DOCKER_BUILDKIT I get nothing. And it still has a huge pause:

python-pull-windows

@daniandl
Copy link

daniandl commented Jul 15, 2021

Didn't work for me. Looks like I wasn't using BuildKit in the first place. If I run echo $DOCKER_BUILDKIT I get nothing. And it still has a huge pause:

I can reproduce this 100%, same issue

@mattwelke
Copy link
Author

mattwelke commented Jul 15, 2021

@daniandl Can you confirm? Is is that you have the same result as @kaner where if you unset the DOCKER_BUILDKIT env var, you don't get the hanging? Or is that you reproduced what I experienced, where you didn't have the env var set in the first place, and your docker pull hung?

@Kleptine
Copy link

I can confirm that I don't have DOCKER_BUILDKIT enabled, and still see the issue.

@liu-xinhui
Copy link

I user centos7.9, and its a Cloud computer, have same issue,load metadata is to slow
image

@djs55
Copy link

djs55 commented Mar 7, 2023

@kakuzei thanks for the update. With 4.17.0, could you verify exactly which docker-credential-desktop.exe file is being executed? Unfortunately in previous releases there were two copies, and one was accidentally unsigned. On my machine I have:

$ which docker-credential-desktop.exe
/usr/bin/docker-credential-desktop.exe
$ ls -l /usr/bin/docker-credential-desktop.exe
lrwxrwxrwx 1 root root 46 Mar  7 14:09 /usr/bin/docker-credential-desktop.exe -> /Docker/host/bin/docker-credential-desktop.exe
$ ls -l /Docker/host/bin/docker-credential-desktop.exe
-rwxrwxrwx 1 root root 14121984 Mar  7 14:08 /Docker/host/bin/docker-credential-desktop.exe

where /Docker/host/bin is mounted from the host.

Assuming that looks ok, could you try running

time /Docker/host/bin/docker-credential-wincred.exe list
time /Docker/host/bin/docker-credential-desktop.exe list

to see if the latency from running docker-credential-wincred.exe is the same as docker-credential-desktop.exe? I'd like to understand if there's a problem with a specific binary or with all binaries. We're investigating the possibility of removing docker-credential-desktop.exe and relying on docker-credential-wincred.exe, so I want to check whether using "credStore": "wincred.exe" would be an improvement or not.

FYI I did a few more experiments with my .docker/config.json. I observed that the CLI does not validate the credStore field, and if it can't find the helper then it silently tries to authenticate without it. I tried pulling a private image from Docker Hub with different values and determined that

  • "credStore": "nothing": fails with an authentication error, i.e. helper was ignored
  • "credStore": "desktop": fails with an authentication error, i.e. helper was ignored because it was missing the .exe suffix
  • "credStore": "desktop.exe": worked
  • "credStore": "wincred.exe": worked
  • "credStore": "wincred": fails with an authentication error, i.e. helper was ignored because it was missing the .exe suffix

I mention this because it's quite confusing behaviour when testing.

@kakuzei
Copy link

kakuzei commented Mar 8, 2023

@djs55 thank you for your answer.

Here are the paths of the docker-credential commands:

$ which docker-credential-wincred.exe
/mnt/c/Program Files/Docker/Docker/resources/bin/docker-credential-wincred.exe

$ which docker-credential-desktop.exe
/usr/bin/docker-credential-desktop.exe

$ ls -l /usr/bin/docker-credential-desktop.exe
-r-xr-xr-x 1 root root 9740640 Feb  3 11:06 /usr/bin/docker-credential-desktop.exe

Please find here the output of some commands when credsStore is set to desktop.exe:

$ time /Docker/host/bin/docker-credential-wincred.exe list
{"https://index.docker.io/v1/":"<<myaccountname>>","https://index.docker.io/v1//access-token":"<<myaccountname>>","https://index.do
cker.io/v1//refresh-token":"<<myaccountname>>"}

real    0m0.153s
user    0m0.002s
sys     0m0.000s

$ time /Docker/host/bin/docker-credential-desktop.exe list
{"https://index.docker.io/v1/":"<<myaccountname>>","https://index.docker.io/v1//access-token":"<<myaccountname>>","https://index.do
cker.io/v1//refresh-token":"<<myaccountname>>"}

real    0m2.922s
user    0m0.002s
sys     0m0.000s

$ time docker-credential-wincred.exe list
{"https://index.docker.io/v1/":"<<myaccountname>>","https://index.docker.io/v1//access-token":"<<myaccountname>>","https://index.do
cker.io/v1//refresh-token":"<<myaccountname>>"}

real    0m0.117s
user    0m0.004s
sys     0m0.000s

$ time docker-credential-desktop.exe list
{"https://index.docker.io/v1/":"<<myaccountname>>","https://index.docker.io/v1//access-token":"<<myaccountname>>","https://index.do
cker.io/v1//refresh-token":"<<myaccountname>>"}

real    4m17.253s
user    0m0.001s
sys     0m0.000s

The latest command executed shows the issue.

@djs55
Copy link

djs55 commented Mar 13, 2023

@kakuzei thanks for the detailed update. On my machine I notice that

# ls -l /usr/bin/docker-credential-desktop.exe
lrwxrwxrwx 1 root root 46 Mar 13 17:34 /usr/bin/docker-credential-desktop.exe -> /Docker/host/bin/docker-credential-desktop.exe

Could you try deleting /usr/bin/docker-credential-desktop.exe and restarting Docker? Hopefully it will create a symlink to the working version of the credential helper.

@kakuzei
Copy link

kakuzei commented Mar 14, 2023

@djs55 Many thanks for that: it solves the 5 minutes delay issue.

Could you confirm me the buggy /usr/bin/docker-credential-desktop.exe file was created by a version of Docker Desktop previous than 4.17.0 ?

@viceice
Copy link

viceice commented Mar 23, 2023

it seems v4.17.1 fixed the delay for me

@chanpark
Copy link

chanpark commented Mar 23, 2023

it seems v4.17.1 fixed the delay for me

👍

I also did the following after uninstalling v4.16.2, before installing v4.17.1. Just in case you still see the problem with 4.17.1, you could try this out.

wsl --unregister docker-desktop
wsl --unregister docker-desktop-data

@docker-robot
Copy link

docker-robot bot commented Jun 23, 2023

There hasn't been any activity on this issue for a long time.
If the problem is still relevant, mark the issue as fresh with a /remove-lifecycle stale comment.
If not, this issue will be closed in 30 days.

Prevent issues from auto-closing with a /lifecycle frozen comment.

/lifecycle stale

@viceice
Copy link

viceice commented Jun 23, 2023

no

@Xun66
Copy link

Xun66 commented Jun 27, 2023

I'm facing the same problem. It always takes 31-32 seconds to "load metadata".

When I enabled debug logging for the docker daemon, I noticed that it got stuck while requesting one of my registry-mirrors (not the first one, though). I tried using curl to test the mirror site and found out that it was actually down. After removing that mirror, the build process became super fast again.

@NickMoignard
Copy link

Yeah still an issue for me

@withinboredom
Copy link

same, still an issue.

@Saibamen
Copy link

Saibamen commented Oct 11, 2023

This same:

Docker Desktop 4.24.1 (Docker version 24.0.6, build ed223bc)
Windows 10 22H2

About 80 seconds for docker.io/trafex/php-nginx:3.4.0 image...

@miparnisari
Copy link

miparnisari commented Jan 15, 2024

I'm experiencing this issue on my Macbook Pro M2.

$ gh repo clone openfga/openfga
$ docker-compose up --build
[+] Building 147.5s (16/17)                                docker:desktop-linux
 => [migrate internal] load build definition from Dockerfile              10.0s
 => => transferring dockerfile: 1.20kB                                    10.0s
 => [openfga internal] load metadata for ghcr.io/grpc-ecosystem/grpc-hea  35.3s
 => [openfga internal] load metadata for cgr.dev/chainguard/go:1.21@sha2  21.1s
 => [openfga internal] load metadata for cgr.dev/chainguard/static@sha25  21.1s
 => [migrate builder 1/4] FROM cgr.dev/chainguard/go:1.21@sha256:60abea54  0.0s
 => [migrate stage-1 1/3] FROM cgr.dev/chainguard/static@sha256:fd59d1089  0.0s
 => [migrate] FROM ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.23@sha25  0.0s
 => [migrate internal] load build context                                 10.4s
 => => transferring context: 725.54kB                                     10.4s
 => CACHED [migrate builder 2/4] WORKDIR /app                              0.0s
 => CACHED [migrate builder 3/4] RUN --mount=type=cache,target=/root/go/p  0.0s
 => [migrate builder 4/4] RUN --mount=type=cache,target=/root/.cache/go-b  1.3s
 => CACHED [migrate stage-1 2/3] COPY --from=ghcr.io/grpc-ecosystem/grpc-  0.0s
 => [migrate stage-1 3/3] COPY --from=builder /bin/openfga /openfga        0.0s
 => [migrate] exporting to image                                           0.1s
 => => exporting layers                                                    0.1s
 => => writing image sha256:be839d2f884ebfe6dec3b13ed1d302077f5b86401a577  0.0s
 => => naming to docker.io/library/openfga-migrate                         0.0s
 => [openfga internal] load .dockerignore                                 10.0s
 => => transferring context: 125B                                         10.0s
 => [openfga internal] load build definition from Dockerfile              10.0s
 => => transferring dockerfile: 1.20kB                                    10.0s

Relevant: I'm on a Wi-Fi hotspot at the airport, sharing from my own phone.

@ericslandry
Copy link

I had a similar issue that I boiled down to this command.

$ docker run debian:bullseye bash -c "apt update >/dev/null 2>&1 && apt install -y dnsutils >/dev/null 2>&1 && time nslookup api.sdkman.io"
Server:         192.168.65.7
Address:        192.168.65.7#53

Non-authoritative answer:
Name:   api.sdkman.io
Address: 45.55.42.78

real    0m11.157s
user    0m0.000s
sys     0m0.025s

You'll notice the nslookup call taking over 11 seconds.

When specifying the quad-8 DNS server in the docker run command , the problem goes away.

 docker run --dns 8.8.8.8 debian:bullseye bash -c "apt update >/dev/null 2>&1 && apt install -y dnsutils >/dev/null 2>&1 && time nslookup api.sdkman.io"
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   api.sdkman.io
Address: 45.55.42.78

real    0m0.094s
user    0m0.004s
sys     0m0.004s

The equivalent behavior can be achieved by setting the DNS in the daemon.json (Docker Desktop | Settings | Docker Engine).

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "dns": [
    "8.8.8.8"
  ],
  "experimental": false
}

My setup: Windows 11, Docker Desktop 4.27.0-135262, Engine 25.0.1

@AnrDaemon
Copy link

Following the advice in https://stackoverflow.com/a/70483395/1449366 and just pulled the problematic images manually.
From now on, the builds are lightning fast. (And no, I did not re-tag them, just did docker pull ….)

@ThePlenkov
Copy link

Does anybody know what triggers ~/.docker/config.json file appearing back after deletion? Thank you!

@ThePlenkov
Copy link

BTW I was not able to solve this problem by removing ~/.docker/config.json because it's recreated on Docker restart, but changing desktop.exe to wincred solved the issue luckily:

{
  "credsStore": "wincred"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests