Docker build: 90s delay resolving local images

[JrDw0]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Every time I build a Docker image that uses ANY localhost/ base image, it takes exactly 90 seconds to resolve the image during the build process, even though the images are already cached locally. This happens with every single localhost image in my environment.

Expected behaviour

Local images should resolve normaly, especially when using --pull=false.

Actual behaviour

Building with any localhost base image always takes 90+ seconds on the resolve step. The image shows as CACHED but still takes 90 seconds to resolve. This is consistent across ALL localhost images in my environment.

Buildx version

github.com/docker/buildx v0.23.0-desktop.1 503f948aadbddb6de3ec5581f766e1d27f6975a1

Docker info

docker info        10:31:18 
Client:
 Version:    28.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.1.7
    Path:     /Users/jrd/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.23.0-desktop.1
    Path:     /Users/jrd/.docker/cli-plugins/docker-buildx
  cloud: Docker Cloud (Docker Inc.)
    Version:  v0.3.0
    Path:     /Users/jrd/.docker/cli-plugins/docker-cloud
  compose: Docker Compose (Docker Inc.)
    Version:  v2.36.0-desktop.1
    Path:     /Users/jrd/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.38
    Path:     /Users/jrd/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.1.8
    Path:     /Users/jrd/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/jrd/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/jrd/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/jrd/.docker/cli-plugins/docker-init
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  dev
    Path:     /Users/jrd/.docker/cli-plugins/docker-mcp
  model: Docker Model Runner (Docker Inc.)
    Version:  v0.1.23
    Path:     /Users/jrd/.docker/cli-plugins/docker-model
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/jrd/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.18.1
    Path:     /Users/jrd/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 9
 Server Version: 28.1.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 17.54GiB
 Name: docker-desktop
 ID: c3e3a7c8-8dbb-45af-943e-2742b8004607
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 62
  Goroutines: 83
  System Time: 2025-06-06T02:31:22.386485134Z
  EventsListeners: 12
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/jrd/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

docker buildx ls
NAME/NODE           DRIVER/ENDPOINT     STATUS    BUILDKIT   PLATFORMS
default             docker
 \_ default          \_ default         running   v0.21.0    linux/amd64 (+2), linux/arm64, linux/ppc64le, linux/s390x, (2 more)
desktop-linux*      docker

Configuration

FROM localhost/openresty-alpine:1.19.9.1
ADD dist.tar.gz /opt/analytics-waterbear/v3/
RUN echo "" > /etc/nginx/conf.d/default.conf && sed -i s#1024#10240# /usr/local/openresty/nginx/conf/nginx.conf

Build command:

$ docker build --pull=false --platform=linux/arm64 -t analytics/analytics-web:1.0.0 -f Dockerfile-prod .

This happens with any localhost image:

FROM localhost/openresty-alpine:1.19.9.1
FROM localhost/any-other-image:tag

Build logs

[INFO] 2025-06-06 09:46:18 Target image does not exist, building: analytics/analytics-web:1.0.0
[INFO] 2025-06-06 09:46:18 Starting image build: analytics/analytics-web:1.0.0
[INFO] 2025-06-06 09:46:18 Dockerfile: Dockerfile-prod, context: .
[INFO] 2025-06-06 09:46:18 Checking base image dependencies...
[INFO] 2025-06-06 09:46:18 Detected base image: localhost/openresty-alpine:1.19.9.1
[INFO] 2025-06-06 09:46:18 Base image exists locally: localhost/openresty-alpine:1.19.9.1
[INFO] 2025-06-06 09:46:18 Base image found locally, skipping pull: localhost/openresty-alpine:1.19.9.1
[COMMAND] 2025-06-06 09:46:18 Executing command: docker build --pull=false --platform=linux/arm64 -t analytics/analytics-web:1.0.0 -f Dockerfile-prod .
[+] Building 90.8s (8/8) FINISHED                                                     docker:desktop-linux
 => [internal] load build definition from Dockerfile-prod                                                     0.0s
 => => transferring dockerfile: 432B                                                                          0.0s
 => [internal] load metadata for localhost/openresty-alpine:1.19.9.1                                         0.0s
 => [internal] load .dockerignore                                                                             0.0s
 => => transferring context: 2B                                                                               0.0s
 => [internal] load build context                                                                             0.1s
 => => transferring context: 2.07MB                                                                           0.0s
 => CACHED [1/3] FROM localhost/openresty-alpine:1.19.9.1@sha256:1998cfeaf36e3d2f57709d537e2eacba1611e2394d528b70b5ff91c492da  90.1s
 => => resolve localhost/openresty-alpine:1.19.9.1@sha256:1998cfeaf36e3d2f57709d537e2eacba1611e2394d528b70b5ff91c492da  90.1s
 => [2/3] ADD dist.tar.gz  /opt/analytics-waterbear/v3/                                                       0.1s
 => [3/3] RUN echo "" > /etc/nginx/conf.d/default.conf && sed -i s#1024#10240# /usr/local/openresty/nginx/conf/nginx.co  0.2s


The image is clearly marked as CACHED but the resolve step still takes 90.1 seconds.

Additional info

Environment:

• macOS M3
• Docker Desktop 28.1.1, build 4eba377
• Docker Compose v2.35.1-desktop.1

What I've verified:

• docker images shows the base image exists locally
• Used --pull=false - same delay
• Cleared Docker cache with docker system prune -a - still happens
• Restarted Docker Desktop - no change
• Updated to latest Docker Desktop - problem persists
• Happens even when offline

Image verification:

$ docker images localhost/openresty-alpine:1.19.9.1
REPOSITORY                TAG       IMAGE ID       CREATED       SIZE
localhost/openresty-alpine   1.19.9.1   abc123...      2 days ago    45.2MB

This affects ALL my localhost images - I have about 10+ different localhost images and every single one has this exact same 90-second delay. It's not specific to the openresty image shown above. The image is definitely there, but Docker seems to be doing some kind of network lookup anyway during the resolve phase.

[tonistiigi]

I'm surprised this works at all. I would have guessed localhost/openresty-alpine gets resolved to docker.io/localhost/openresty-alpine because there is no clear domain name (but I guess there is some special condition somewhere).

There is some kind of timeout somewhere. You can look if docker buildx history trace shows something better. I couldn't repro. Even without --pull though this shouldn't resolve anything.

» docker buildx build .                                                                                                                                                                    !7666
[+] Building 0.3s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                        0.0s
 => => transferring dockerfile: 150B                                                                                                                                                        0.0s
 => [internal] load metadata for localhost/openresty-alpine:1.19.9.1                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                           0.0s
 => => transferring context: 2B                                                                                                                                                             0.0s
 => [internal] load build context                                                                                                                                                           0.0s
 => => transferring context: 150B                                                                                                                                                           0.0s
 => CACHED [1/3] FROM localhost/openresty-alpine:1.19.9.1                                                                                                                                   0.0s
 => [2/3] ADD Dockerfile /opt/analytics-waterbear/v3/                                                                                                                                       0.0s
 => [3/3] RUN echo "" > /default.conf                                                                                                                                                       0.1s
 => exporting to image                                                                                                                                                                      0.0s
 => => exporting layers                                                                                                                                                                     0.0s
 => => writing image sha256:b43bd0ab0ecb85f4ffac7c573ba42f1f234b83f97c9084497f18db30cfaf79aa                                                                                                0.0s

[tonistiigi]

Ok, I think this is related to containerd image store. I can reproduce then. Seems to be specific to "localhost", other names work, as does 127.0.0.1.

[JrDw0]

I change localhost to local, the problem still exists.

I use docker buildx history trace

How should I save my wasted time and life :(

[thaJeztah]

I'm surprised this works at all. I would have guessed localhost/openresty-alpine gets resolved to docker.io/localhost/openresty-alpine because there is no clear domain name (but I guess there is some special condition somewhere).

Yes, there's some special cases; https://github.com/distribution/reference/blob/ff14fafe2236e51c2894ac07d4bdfc778e96d682/regexp.go#L55-L58

[tonistiigi]

In the screenshot you can see that it is pulling image from docker.io/local/* what would be correct if you didn't have local image with exactly the same name. If you do have such image then I couldn't reproduce. The screenshot is cut off and it is not visible where the timeout is. Just that cache request took minute and the HTTP request only took 400ms. If the gap is in between 401 and next authenticated requests then the actual delay could likely be in the credential helper configuration.

[JrDw0]

In the screenshot you can see that it is pulling image from docker.io/local/* what would be correct if you didn't have local image with exactly the same name. If you do have such image then I couldn't reproduce. The screenshot is cut off and it is not visible where the timeout is. Just that cache request took minute and the HTTP request only took 400ms. If the gap is in between 401 and next authenticated requests then the actual delay could likely be in the credential helper configuration.

Here is the complete screenshot, which contains time and exception information, but it didn't help me solve this problem.

Could you please help me analyze the specific reasons behind this exception based on the information in the screenshot? I would greatly appreciate any guidance or suggestions you might have.

[JrDw0]

I tried setting DOCKER_BUILDKIT=0, and this timeout didn't occur.

By the way, it may be because I'm in China, and the network environment has caused a timeout when connecting to the Docker Hub authentication server. But I wonder if there is a way for me to skip this step?

docker build  --platform=linux/arm64 -t test -f Dockerfile-prod .                     
[+] Building 60.4s (10/10) FINISHED                                                 docker:desktop-linux
 => [internal] load build definition from Dockerfile-prod                                           0.0s
 => => transferring dockerfile: 415B                                                                0.0s
 => [internal] load metadata for docker.io/library/openresty-alpine:1.19.9.1                        0.0s
 => [internal] load .dockerignore                                                                   0.0s
 => => transferring context: 2B                                                                     0.0s
 => [internal] load build context                                                                   0.0s
 => => transferring context: 35B                                                                    0.0s
 => [1/3] FROM docker.io/library/openresty-alpine:1.19.9.1@sha256:1998cfeaf36e3d2f57709d537e2eacb  60.1s
 => => resolve docker.io/library/openresty-alpine:1.19.9.1@sha256:1998cfeaf36e3d2f57709d537e2eacb  60.1s
 => [auth] library/openresty-alpine:pull token for registry-1.docker.io                             0.0s
 => [auth] library/openresty-alpine:pull token for registry-1.docker.io                             0.0s
 => CACHED [2/3] ADD dist.tar.gz  /opt/waterbear-monitor                                            0.0s
 => CACHED [3/3] RUN echo "" > /etc/nginx/conf.d/default.conf && sed -i s#1024#10240# /usr/local/o  0.0s
 => exporting to image                                                                              0.1s
 => => exporting layers                                                                             0.0s
 => => exporting manifest sha256:08b892f3d53ee90f12db3a8508f4e4a0ceee114a26c83233a06451fbb0d44831   0.0s
 => => exporting config sha256:66d746607855bf1636b761eb4132fc02d5210d4adef2bf30c4d8b43adb167076     0.0s
 => => exporting attestation manifest sha256:446736d30edfa072ce94fa0f3b7685868ea058f6b221e8d98c808  0.0s
 => => exporting manifest list sha256:b18813d2f659a40da229f54a6a443ba7c1a567989f01e8308f94783d4964  0.0s
 => => naming to docker.io/library/test:latest                                                      0.0s
 => => unpacking to docker.io/library/test:latest                                                   0.0s


 DOCKER_BUILDKIT=0  docker build  --platform=linux/arm64 -t test -f Dockerfile-prod .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  383.9MB
Step 1/4 : FROM openresty-alpine:1.19.9.1
 ---> 1998cfeaf36e
Step 2/4 : ENV TZ=Asia/Shanghai
 ---> Using cache
 ---> 8dab9426ae51
Step 3/4 : ADD dist.tar.gz  /opt/waterbear-monitor
 ---> Using cache
 ---> d830ae851158
Step 4/4 : RUN echo "" > /etc/nginx/conf.d/default.conf && sed -i s#1024#10240# /usr/local/openresty/nginx/conf/nginx.conf && sed -i '25i\worker_rlimit_nofile 65535;' /usr/local/openresty/nginx/conf/nginx.conf
 ---> Using cache
 ---> 0b78f1cd59c4
Successfully built 0b78f1cd59c4
Successfully tagged test:latest

[tonistiigi]

This was fixed in moby/buildkit#6029