Buildx doesn't respect insecure registry configuration

[Re4zOon]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Hi,

Re-opening 2226, as the issue still exists.
See comments in the other ticket.
The attached logs/outputs are fresh.

Expected behaviour

Use port 80

Actual behaviour

Uses port 443

Buildx version

github.com/docker/buildx v0.17.1 257815a

Docker info

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
WARNING: Plugin "/usr/libexec/docker/cli-plugins/docker-buildx-014.bak" is not valid: plugin candidate "buildx-014.bak" did not match "^[a-z][a-z0-9]*$"

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 29
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-503.14.1.el9_5.x86_64
 Operating System: Red Hat Enterprise Linux 9.5 (Plow)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.37GiB
 Name: hostname
 ID: 2864ada1-48b8-44ae-8094-dc18a3baed6c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: username
 Experimental: false
 Insecure Registries:
  our-registry
  127.0.0.0/8
 Registry Mirrors:
  http://our-registry/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.31.0.0/16, Size: 24

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Builders list

NAME/NODE         DRIVER/ENDPOINT                   STATUS     BUILDKIT   PLATFORMS
default           docker
 \_ default        \_ default                       running    v0.16.0    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

Configuration

FROM our-registry/base-images/debian:latest

RUN echo "hello"

Build logs

[root@sgujenkins2 ~]# docker build .
[+] Building 0.3s (2/2) FINISHED                                                                         docker:default
 => [internal] load build definition from Dockerfile                                                               0.1s
 => => transferring dockerfile: 112B                                                                               0.0s
 => ERROR [internal] load metadata for our-registry/base-images/debian:latest                         0.0s
------
 > [internal] load metadata for our-registry/base-images/debian:latest:
------
Dockerfile:1
--------------------
   1 | >>> FROM our-registry/base-images/debian:latest
   2 |
   3 |     RUN echo "hello"
--------------------
ERROR: failed to solve: our-registry/base-images/debian:latest: failed to resolve source metadata for our-registry/base-images/debian:latest: failed to do request: Head "https://our-registry/v2/base-images/debian/manifests/latest": dial tcp 10.0.0.1:443: connect: no route to host

Additional info

❯ cat /etc/docker/daemon.json
{
"insecure-registries":["http://our-registry", "our-registry"],
"registry-mirrors": ["http://our-registry"]
}

[Kraust]

Have you tried removing the running docker buildx container docker buildx rm? That is what I believe solved the issue for me.

[Re4zOon]

Have you tried removing the running docker buildx container docker buildx rm? That is what I believe solved the issue for me.

Hi @Kraust,

As you see I only have one builder, which is the default (not even a container).
The weirdest thing is the "no route to host", which is incorrect, as with the legacy builder (on the same URL/IP) I can connect.

Anyways, this happen from all multiple of our systems, not just one (even from every DEV local WSL/Ubuntu).

[tonistiigi]

From the error it seems to be like network route error, not "insecure registry" config error. Insecure registries in dockerd mean both HTTP and HTTPS with insecure registry.

[Re4zOon]

Hi,

There is no 443 port open on the registry.
I never wanted docker to try 443 (https).
However it still does, which is not the same behaviour as legacy builder.
export DOCKER_BUILDKIT=0 "solves" the issue (as in it wont use buildx/this repo).

[uliss3s]

Same issue: #2712

Same temp solution: export DOCKER_BUILDKIT=0

[martadinata666]

$ docker buildx create --use --driver-opt env.BUILDKIT_UNSAFE_HTTP=1 --config buildkit-config.toml
nice_germain
$ docker info
Client:
 Version:    28.2.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.25.0
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
Server:
 Containers: 64
  Running: 61
  Paused: 0
  Stopped: 3
 Images: 55
 Server Version: 27.5.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: fluentd
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local local-persist
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: r86fuksfkz1m8cpbnpp5pkbh0
  Is Manager: true
  ClusterID: tokof9pri5nwl34mqw6y5w2b9
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.0.2
  Manager Addresses:
   192.168.0.2:2377
 Runtimes: youki crun io.containerd.runc.v2 runc
 Default Runtime: crun
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: 9c9a76ac11994701dd666c4f0b869ceffb599a66
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.12-rt-x64v3-xanmod1
 Operating System: Ubuntu 24.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 31.01GiB
 Name: homelab
 ID: 2JTD:RTQI:7ACK:663S:F2DR:T5MR:BAHF:VMYN:WOAE:WWVC:HHVS:KCSN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  192.168.0.2:5050
  192.168.0.2:6050
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/16, Size: 16
   Base: 172.18.0.0/16, Size: 16
   Base: 172.19.0.0/16, Size: 16
   Base: 172.20.0.0/16, Size: 16
   Base: 172.21.0.0/16, Size: 16
   Base: 172.22.0.0/16, Size: 16
   Base: 172.23.0.0/16, Size: 16
   Base: 172.24.0.0/16, Size: 16
   Base: 172.25.0.0/16, Size: 16
   Base: 172.26.0.0/16, Size: 16
   Base: 172.27.0.0/16, Size: 16
   Base: 172.28.0.0/16, Size: 16
   Base: 172.29.0.0/16, Size: 16
   Base: 172.[30](http://192.168.0.2:10000/dedyms/iperf3/-/jobs/4687#L30).0.0/16, Size: 16
   Base: 192.168.0.0/16, Size: 20
[DEPRECATION NOTICE]: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
In future versions this will be a hard failure preventing the daemon from starting! Learn more at: https://docs.docker.com/go/api-security/
$ docker buildx version
github.com/docker/buildx v0.25.0 faaea65da4ba0e58a13cd9cadcb950c51cf3b3c9
$ docker buildx inspect --bootstrap
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 2.4s done
#1 creating container buildx_buildkit_nice_germain0
#1 creating container buildx_buildkit_nice_germain0 1.0s done
#1 DONE 3.4s
Name:          nice_germain
Driver:        docker-container
Last Activity: 2025-06-22 03:08:42 +0000 UTC
Nodes:
Name:                  nice_germain0
Endpoint:              unix:///var/run/docker.sock
Driver Options:        env.BUILDKIT_UNSAFE_HTTP="1"
Status:                running
BuildKit daemon flags: --allow-insecure-entitlement=network.host
BuildKit version:      v0.22.0
Platforms:             linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/loong64, linux/arm/v7, linux/arm/v6
Labels:
 org.mobyproject.buildkit.worker.executor:         oci
 org.mobyproject.buildkit.worker.hostname:         bfd7d51c4505
 org.mobyproject.buildkit.worker.network:          host
 org.mobyproject.buildkit.worker.oci.process-mode: sandbox
 org.mobyproject.buildkit.worker.selinux.enabled:  false
 org.mobyproject.buildkit.worker.snapshotter:      overlayfs
GC Policy rule#0:
 All:            false
 Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration:  48h0m0s
 Max Used Space: 488.3MiB
GC Policy rule#1:
 All:            false
 Keep Duration:  1440h0m0s
 Reserved Space: 9.[31](http://192.168.0.2:10000/dedyms/iperf3/-/jobs/4687#L31)3GiB
 Max Used Space: 81.96GiB
 Min Free Space: 20.49GiB
GC Policy rule#2:
 All:            false
 Reserved Space: 9.313GiB
 Max Used Space: 81.96GiB
 Min Free Space: 20.49GiB
GC Policy rule#3:
 All:            true
 Reserved Space: 9.313GiB
 Max Used Space: 81.96GiB
 Min Free Space: 20.49GiB
File#buildkitd.toml:
 > 
 > [registry]
 > 
 >   [registry."192.168.0.2:5050"]
 >     http = true
 >     insecure = true
 > 
$ docker buildx build --push --platform "$BUILD_PLATFORM" -t "$CI_REGISTRY_IMAGE" -t "$CI_REGISTRY_IMAGE:$RELEASE" . -f Dockerfile
#0 building with "nice_germain" instance using docker-container driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 253B done
#1 DONE 0.0s
#2 [auth] dedyms/debian:pull token for 192.168.0.2:5050
#2 DONE 0.0s
#3 [internal] load metadata for 192.168.0.2:5050/dedyms/debian:latest
#3 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://192.168.0.2:5050/v2/dedyms/debian/blobs/sha256:f69545bb9238[32](http://192.168.0.2:10000/dedyms/iperf3/-/jobs/4687#L32)6a590ac3a7663a6faf5e40193e655cbd11a0492490ab130c12: 400 Bad Request

The same config works on v0.12.2, people said export this and that but nothing works, only works when using old version v0.12.2(latest known works)

[alexandergunnarson]

What solved the issue for me was

config_file=$(mktemp)
cat > "$config_file" << EOF
[registry."127.0.0.1:5000"]
http = true
insecure = true
EOF

docker buildx create \
    --name "$name" \
    --driver docker-container \
    --driver-opt env.BUILDKIT_UNSAFE_HTTP=1 \
    --driver-opt network=host \
    --config "$config_file" \
    --use

specifically --driver-opt network=host was the missing piece for me.

FWIW:

Client: Docker Engine - Community
 Version:    28.3.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.25.0