can't pull images while building with buildx 'failed to do request', 'i/o timeout'

[xonq]

PROBLEM
when trying to pull any image in the Dockerfile, buildx build fails. I can pull images manually (docker pull), wouldn't mind just using my local version, but it preferentially pulls from the website for whatever reason. Not behind a proxy afaik.

SETUP
Added buildx binary manually - worth noting I couldn't use buildx simply by specifying export DOCKER_CLI_EXPERIMENTAL=enabled.
Arch Linux
Docker version 19.03.5-ce, build 633a0ea838

HIGHLIGHTED ERROR
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: failed to do request: Head https://registry-1.docker.io/v2/continuumio/miniconda/manifests/latest: dial tcp: lookup registry-1.docker.io on 192.168.1.1:53: read udp 172.17.0.2:40167->192.168.1.1:53: i/o timeout

LOGS
docker buildx build --platform linux/amd64,linux/arm64 --output /home/xaq/software/maker.img .

 => [internal] load build definition from Dockerfile                                                                             0.0s
 => => transferring dockerfile: 32B                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                0.0s
 => => transferring context: 2B                                                                                                  0.0s
 => ERROR [linux/amd64 internal] load metadata for docker.io/continuumio/miniconda:latest                                       10.0s
 => ERROR [linux/arm64 internal] load metadata for docker.io/continuumio/miniconda:latest                                       10.0s
 => ERROR [linux/arm64 1/5] FROM docker.io/continuumio/miniconda:latest                                                         10.0s
 => => resolve docker.io/continuumio/miniconda:latest                                                                           10.0s
 => ERROR [linux/amd64 1/5] FROM docker.io/continuumio/miniconda:latest                                                         10.0s
 => => resolve docker.io/continuumio/miniconda:latest                                                                           10.0s
 => [internal] load build context                                                                                                0.0s
------
 > [linux/amd64 internal] load metadata for docker.io/continuumio/miniconda:latest:
------
------
 > [linux/arm64 internal] load metadata for docker.io/continuumio/miniconda:latest:
------
------
 > [linux/arm64 1/5] FROM docker.io/continuumio/miniconda:latest:
------
------
 > [linux/amd64 1/5] FROM docker.io/continuumio/miniconda:latest:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: failed to do request: Head https://registry-1.docker.io/v2/continuumio/miniconda/manifests/latest: dial tcp: lookup registry-1.docker.io on 192.168.1.1:53: read udp 172.17.0.2:40167->192.168.1.1:53: i/o timeout

Dockerfile:

FROM continuumio/miniconda

WORKDIR /home/xaq/software

COPY /home/xaq/software/environment.yml ./

RUN conda env create -f environment.yml

RUN echo "source activate maker" > ~/.bashrc
ENV PATH /opt/conda/envs/maker/bin:$PATH

docker buildx ls

NAME/NODE    DRIVER/ENDPOINT             STATUS  PLATFORMS
mybuilder *  docker-container                    
mybuilder0 unix:///var/run/docker.sock running linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

default      docker                              
  default    default                     running linux/amd64, linux/386

[xonq]

Unlike #136, I'm not using a proxy. echo "$http_proxy" yields no results.
I tried to use this build config, but it's failing:

 => ERROR [internal] booting buildkit                                                                                            2.9s
 => => starting container buildx_buildkit_mybuilder0                                                                             2.9s
------
 > [internal] booting buildkit:
------
Name:   mybuilder
Driver: docker-container

Nodes:
Name:      mybuilder0
Endpoint:  unix:///var/run/docker.sock
Status:    stopped
Platforms: 

[tonistiigi]

What is your dns configuration in this machine? Does networking work with other docker containers with this configuration.

[xonq]

Don't have any DNS settings on /etc/hosts and /etc/resolv.conf; my router is automatically acquiring that from ISP.

Opened an alpine container and ran a couple ping commands, none worked. ip a is 172.17.0.3 for eth0@if32

Running the laptop as the head node for a small cluster - however dhcp is not active

/etc/hosts/

127.0.0.1       localhost
::1             localhost
192.168.1.15    node00.localdomain      node00
10.254.239.10   node00
10.254.239.11   node01
10.254.239.12   node02
10.254.239.13   node03
10.254.239.14   node04

/etc/resolv.conf
nameserver 192.168.1.1

[xonq]

issue was nftables not allowing ip forwarding, so the docker bridge network was failing.
solved by disabling/adding forwarding rule to firewall

[altrobot]

@xonq I have the same problem. Could you explain more details as to how you handle this issue?

[xonq]

@altrobot If your problem is caused by the same thing as me, then it is due to nftables or iptables blocking buildx traffic. A simple workaround is to stop whichever one you're using, HOWEVER this leaves you vulnerable, so a longterm solution is to find the specific problem in your firewall configuration or implement an exception for buildx which is beyond me.

Short term solution:
systemctl stop iptables && systemctl stop nftables whenever you want to run buildx then reactivate via systemctl start iptables OR systemctl start nftables.

I don't recommend it, but if you constantly need to use buildx you can disable iptables and nftables by replacing stop in the above commands with disable

[frankitox]

Well, in case someone hits this timeout problem when using buildx in WSL2, here's a possible explanation and solution:

Check that WSL's eth0 and docker0 ip ranges don't collide:

ip a

4: eth0: ...
    inet 172.17.113.103/28 brd 172.17.113.111 scope global eth0
6: docker0: ...
    inet 172.17.0.1/16 brd 172.26.255.255 scope global docker0

If they do collide, then to fix it you can modify /etc/docker/daemon.json to change the default bridge network subnet:

{
  "bip": "172.26.0.1/16"
}

Now restarting docker with sudo service docker restart, will allow you to use buildx.

[polarathene]

I didn't want to mess with iptables / nftables but I was getting a similar error with "no such host".

I was using the docker-container driver and found that if you create a builder with the --driver-opt network=host, it'll no longer have an issue pulling images (or presumably any external request?) during a build.

[enescakir]

We got same error. We found out docker containers can't resolve DNS by default.
If you don't want share host network with container for security reason, updating container's /etc/resolv.conf file fixed our problem.

You can also add DNS to docker daemon config, so all new containers have new dns config.

Add following config to /etc/docker/daemon.json

{
    "dns": ["8.8.8.8"]
}

Then restart docker sudo systemctl restart docker

[sky92zwq]

how to use the image I have already pulled by docker ? I want to use docker buildx in my network isolation machine.