Throw error while merging manifest list

[purebaba]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

When I'm building multi-platform images, I encounter an error.

Builder create command is:
docker buildx create --bootstrap --driver=docker-container --name ec4 --config /etc/buildkitd.toml --platform linux/arm64 ssh://root@47.236.8.180
docker buildx create --append --bootstrap --driver=docker-container --name ec4 --config /etc/buildkitd.toml --platform linux/amd64 default

buildkitd.toml file

debug = true
insecure-entitlements = [ "network.host", "security.insecure" ]

[registry."mydomain.com"]
  insecure = true

The build command is:
docker buildx build -t [mydomain.com]/library/node:lts-ubuntu-test --platform linux/arm64,linux/amd64 --push -f Dockerfile_node_lts_ubuntu .

And the images have already been pushed to Harbor.

Expected behaviour

When I tag my image repository as insecure=true, I believe certificate validation should be skipped.

Actual behaviour

failed open: failed to do request: Get "https://registry-i.ezbim.net:8443/v2/library/node/manifests/sha256:e644e02ddf50541389fd2d79e12779720abed6f8b56976769f66e3c305ef1b40": tls: failed to verify certificate: x509: certificate signed by unknown authority

Buildx version

github.com/docker/buildx v0.11.2 9872040

Docker info

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 4
  Running: 1
  Paused: 0
  Stopped: 3
 Images: 7
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.4.254-1.el7.elrepo.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.765GiB
 Name: 192-168-1-206.ezbim.net
 ID: 0aca06b4-b6de-42d3-b45f-c122e22ae9da
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  registry-i.ezbim.net:8443
  registry.ezbim.net
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

ec4 *     docker-container                                         
  ec40    ssh://root@47.236.8.180     running v0.12.1              linux/arm64*, linux/arm/v7, linux/arm/v6
  ec41    unix:///var/run/docker.sock running v0.12.1              linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default   docker                                                   
  default default                     running v0.11.6+0a15675913b7 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Configuration

FROM ubuntu:latest

RUN groupadd --gid 102 bim
&& useradd --uid 102 --gid bim --shell /bin/bash --create-home bim

ARG tag
ENV VERSION ${tag}
ENV NODE_VERSION 16.13.1
ENV LANG C.UTF-8

RUN ARCH= && dpkgArch="$(dpkg --print-architecture)"
&& case "${dpkgArch##-}" in
amd64) ARCH='x64';;
ppc64el) ARCH='ppc64le';;
s390x) ARCH='s390x';;
arm64) ARCH='arm64';;
armhf) ARCH='armv7l';;
i386) ARCH='x86';;
) echo "unsupported architecture"; exit 1 ;;
esac
&& set -ex
# libatomic1 for arm
&& apt-get update && apt-get install -y ca-certificates tzdata curl wget gnupg dirmngr xz-utils libatomic1 --no-install-recommends
#&& apt-get install -y locales
#&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
&& ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
&& dpkg-reconfigure -f noninteractive tzdata
&& rm -rf /var/lib/apt/lists/
&& for key in
4ED778F539E3634C779C87C6D7062848A1AB005C
94AE36675C464D64BAFA68DD7434390BDBE9B9C5
74F12602B6F1C4E913FAA37AD3A89613643B6201
71DCFD284A79C3B38668286BC97EC7A07EDE3FC1
8FCCA13FEF1D0C2E91008E09770F7A9A5AE15600
C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8
C82FA3AE1CBEDC6BE46B9360C43CEC45C17AB93C
DD8F2338BAE7501E3DD5AC78C273792F7D83545D
A48C2BEE680E841632CD4E44F07496B3EB3C1762
108F52B48DB57BB0CC439B2997B01419BD92F80A
B9E2F5981AA6E0CD28160D9FF13993A75599653C
; do
gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$key" ||
gpg --batch --keyserver keyserver.ubuntu.com --recv-keys "$key" ;
done
&& curl -fsSLO --compressed "https://mirrors.tuna.tsinghua.edu.cn/nodejs-release/v$NODE_VERSION/node-v$NODE_VERSION-linux-$ARCH.tar.xz"
&& curl -fsSLO --compressed "https://mirrors.tuna.tsinghua.edu.cn/nodejs-release/v$NODE_VERSION/SHASUMS256.txt.asc"
&& gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc
&& grep " node-v$NODE_VERSION-linux-$ARCH.tar.xz$" SHASUMS256.txt | sha256sum -c -
&& tar -xJf "node-v$NODE_VERSION-linux-$ARCH.tar.xz" -C /usr/local --strip-components=1 --no-same-owner
&& rm "node-v$NODE_VERSION-linux-$ARCH.tar.xz" SHASUMS256.txt.asc SHASUMS256.txt
&& apt-mark auto '.' > /dev/null
&& find /usr/local -type f -executable -exec ldd '{}' ';'
| awk '/=>/ { print $(NF-1) }'
| sort -u
| xargs -r dpkg-query --search
| cut -d: -f1
| sort -u
| xargs -r apt-mark manual
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false
&& apt-get update
&& apt-get install gosu
## canvas依赖
## && apt-get update
## && apt-get install -y build-essential libcairo2-dev libpango1.0-dev libjpeg-dev libgif-dev librsvg2-dev libxinerama1
## && apt-get install -y python3
## && ln -sf /usr/bin/python3 /usr/bin/python
&& ln -s /usr/local/bin/node /usr/local/bin/nodejs
&& apt-get clean && apt-get autoclean && rm -rf /var/lib/apt/lists/*

# smoke tests

RUN node --version
&& npm --version

RUN npm install pm2 -g

Build logs

#0 building with "ec4" instance using docker-container driver

#1 [internal] load build definition from Dockerfile_node_lts_ubuntu
#1 transferring dockerfile: 3.35kB done
#1 DONE 1.4s

#2 [internal] load build definition from Dockerfile_node_lts_ubuntu
#2 transferring dockerfile: 3.35kB 1.7s done
#2 DONE 1.7s

#3 [linux/arm64 internal] load metadata for docker.io/library/ubuntu:latest
#3 DONE 0.9s

#4 [linux/amd64 internal] load metadata for docker.io/library/ubuntu:latest
#4 ...

#5 [internal] load .dockerignore
#5 transferring context: 2B 0.1s done
#5 DONE 0.1s

#4 [linux/amd64 internal] load metadata for docker.io/library/ubuntu:latest
#4 ...

#6 [linux/arm64 1/5] FROM docker.io/library/ubuntu:latest@sha256:ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77
#6 resolve docker.io/library/ubuntu:latest@sha256:ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77 done
#6 DONE 0.0s

#7 [linux/arm64 3/5] RUN ARCH= && dpkgArch="$(dpkg --print-architecture)"     && case "${dpkgArch##*-}" in       amd64) ARCH='x64';;       ppc64el) ARCH='ppc64le';;       s390x) ARCH='s390x';;       arm64) ARCH='arm64';;       armhf) ARCH='armv7l';;       i386) ARCH='x86';;       *) echo "unsupported architecture"; exit 1 ;;     esac     && set -ex     && apt-get update && apt-get install -y ca-certificates tzdata curl wget gnupg dirmngr xz-utils libatomic1 --no-install-recommends     && ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime     && dpkg-reconfigure -f noninteractive tzdata     && rm -rf /var/lib/apt/lists/*     && for key in       4ED778F539E3634C779C87C6D7062848A1AB005C       94AE36675C464D64BAFA68DD7434390BDBE9B9C5       74F12602B6F1C4E913FAA37AD3A89613643B6201       71DCFD284A79C3B38668286BC97EC7A07EDE3FC1       8FCCA13FEF1D0C2E91008E09770F7A9A5AE15600       C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8       C82FA3AE1CBEDC6BE46B9360C43CEC45C17AB93C       DD8F2338BAE7501E3DD5AC78C273792F7D83545D       A48C2BEE680E841632CD4E44F07496B3EB3C1762       108F52B48DB57BB0CC439B2997B01419BD92F80A       B9E2F5981AA6E0CD28160D9FF13993A75599653C     ; do       gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$key" ||       gpg --batch --keyserver keyserver.ubuntu.com --recv-keys "$key" ;     done     && curl -fsSLO --compressed "https://mirrors.tuna.tsinghua.edu.cn/nodejs-release/v$NODE_VERSION/node-v$NODE_VERSION-linux-$ARCH.tar.xz"     && curl -fsSLO --compressed "https://mirrors.tuna.tsinghua.edu.cn/nodejs-release/v$NODE_VERSION/SHASUMS256.txt.asc"     && gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc     && grep " node-v$NODE_VERSION-linux-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c -     && tar -xJf "node-v$NODE_VERSION-linux-$ARCH.tar.xz" -C /usr/local --strip-components=1 --no-same-owner     && rm "node-v$NODE_VERSION-linux-$ARCH.tar.xz" SHASUMS256.txt.asc SHASUMS256.txt     && apt-mark auto '.*' > /dev/null     && find /usr/local -type f -executable -exec ldd '{}' ';'       | awk '/=>/ { print $(NF-1) }'       | sort -u       | xargs -r dpkg-query --search       | cut -d: -f1       | sort -u       | xargs -r apt-mark manual     && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false     && apt-get update     && apt-get install gosu     && ln -s /usr/local/bin/node /usr/local/bin/nodejs     && apt-get clean && apt-get autoclean && rm -rf /var/lib/apt/lists/*
#7 CACHED

#8 [linux/arm64 4/5] RUN node --version     && npm --version
#8 CACHED

#9 [linux/arm64 2/5] RUN groupadd --gid 102 bim   && useradd --uid 102 --gid bim --shell /bin/bash --create-home bim
#9 CACHED

#10 [linux/arm64 5/5] RUN npm install pm2 -g --unsafe-perm --registry=http://npm.ezbim.cn:3038 --verbose
#10 CACHED

#11 exporting to image
#11 exporting layers done
#11 exporting manifest sha256:bb82c7a34f04193e2e2fedde35583d7467df9dd3838a7cfbd636e84064c14f3d done
#11 exporting config sha256:0ac7cca64f23e0f9a3ad192e1a09ac9a9a21271f2e32e7c5e292ef7baeb210b7 done
#11 exporting attestation manifest sha256:4cabaa4badb3beec8e11f717a9cb62e47bcbcc31944b8162ec571d20dc7ab176 0.0s done
#11 exporting manifest list sha256:e644e02ddf50541389fd2d79e12779720abed6f8b56976769f66e3c305ef1b40 done
#11 pushing layers
#11 ...

#4 [linux/amd64 internal] load metadata for docker.io/library/ubuntu:latest
#4 DONE 2.4s

#12 [internal] load .dockerignore
#12 transferring context: 2B done
#12 DONE 0.1s

#13 [linux/amd64 1/5] FROM docker.io/library/ubuntu:latest@sha256:ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77
#13 resolve docker.io/library/ubuntu:latest@sha256:ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77 0.1s done
#13 DONE 0.1s

#11 exporting to image
#11 ...

#14 [linux/amd64 4/5] RUN node --version     && npm --version
#14 CACHED

#15 [linux/amd64 2/5] RUN groupadd --gid 102 bim   && useradd --uid 102 --gid bim --shell /bin/bash --create-home bim
#15 CACHED

#16 [linux/amd64 3/5] RUN ARCH= && dpkgArch="$(dpkg --print-architecture)"     && case "${dpkgArch##*-}" in       amd64) ARCH='x64';;       ppc64el) ARCH='ppc64le';;       s390x) ARCH='s390x';;       arm64) ARCH='arm64';;       armhf) ARCH='armv7l';;       i386) ARCH='x86';;       *) echo "unsupported architecture"; exit 1 ;;     esac     && set -ex     && apt-get update && apt-get install -y ca-certificates tzdata curl wget gnupg dirmngr xz-utils libatomic1 --no-install-recommends     && ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime     && dpkg-reconfigure -f noninteractive tzdata     && rm -rf /var/lib/apt/lists/*     && for key in       4ED778F539E3634C779C87C6D7062848A1AB005C       94AE36675C464D64BAFA68DD7434390BDBE9B9C5       74F12602B6F1C4E913FAA37AD3A89613643B6201       71DCFD284A79C3B38668286BC97EC7A07EDE3FC1       8FCCA13FEF1D0C2E91008E09770F7A9A5AE15600       C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8       C82FA3AE1CBEDC6BE46B9360C43CEC45C17AB93C       DD8F2338BAE7501E3DD5AC78C273792F7D83545D       A48C2BEE680E841632CD4E44F07496B3EB3C1762       108F52B48DB57BB0CC439B2997B01419BD92F80A       B9E2F5981AA6E0CD28160D9FF13993A75599653C     ; do       gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$key" ||       gpg --batch --keyserver keyserver.ubuntu.com --recv-keys "$key" ;     done     && curl -fsSLO --compressed "https://mirrors.tuna.tsinghua.edu.cn/nodejs-release/v$NODE_VERSION/node-v$NODE_VERSION-linux-$ARCH.tar.xz"     && curl -fsSLO --compressed "https://mirrors.tuna.tsinghua.edu.cn/nodejs-release/v$NODE_VERSION/SHASUMS256.txt.asc"     && gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc     && grep " node-v$NODE_VERSION-linux-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c -     && tar -xJf "node-v$NODE_VERSION-linux-$ARCH.tar.xz" -C /usr/local --strip-components=1 --no-same-owner     && rm "node-v$NODE_VERSION-linux-$ARCH.tar.xz" SHASUMS256.txt.asc SHASUMS256.txt     && apt-mark auto '.*' > /dev/null     && find /usr/local -type f -executable -exec ldd '{}' ';'       | awk '/=>/ { print $(NF-1) }'       | sort -u       | xargs -r dpkg-query --search       | cut -d: -f1       | sort -u       | xargs -r apt-mark manual     && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false     && apt-get update     && apt-get install gosu     && ln -s /usr/local/bin/node /usr/local/bin/nodejs     && apt-get clean && apt-get autoclean && rm -rf /var/lib/apt/lists/*
#16 CACHED

#17 [linux/amd64 5/5] RUN npm install pm2 -g --unsafe-perm --registry=http://npm.ezbim.cn:3038 --verbose
#17 CACHED

#18 exporting to image
#18 exporting layers done
#18 exporting manifest sha256:5720b8de405e8a249cbc7d1eb2f307e676f1d818b5bd959e01876c119c15d825 0.0s done
#18 exporting config sha256:a1951416e7c10a0d81399b8609634d4d5a1ad30bfe7cc8c77579c53b6e0521fe 0.0s done
#18 exporting attestation manifest sha256:cb8d7fc7cb8d78fad90f93bbda37299ee246b20f92530fde224e64d2addd7083
#18 exporting attestation manifest sha256:cb8d7fc7cb8d78fad90f93bbda37299ee246b20f92530fde224e64d2addd7083 0.1s done
#18 exporting manifest list sha256:6ccefa1ca04ce8ddadad10711a0cbf537fe5181aeaa2811c6392eba01bfc1f48 0.0s done
#18 pushing layers 1.6s done
#18 pushing manifest for registry-i.ezbim.net:8443/library/node
#18 pushing manifest for registry-i.ezbim.net:8443/library/node 1.0s done
#18 DONE 3.1s

#11 exporting to image
#11 pushing layers 2.6s done
#11 pushing manifest for registry-i.ezbim.net:8443/library/node 1.2s done
#11 DONE 3.8s

#19 merging manifest list registry-i.ezbim.net:8443/library/node:lts-ubuntu-test
#19 ERROR: httpReadSeeker: failed open: failed to do request: Get "https://registry-i.ezbim.net:8443/v2/library/node/manifests/sha256:e644e02ddf50541389fd2d79e12779720abed6f8b56976769f66e3c305ef1b40": tls: failed to verify certificate: x509: certificate signed by unknown authority
------
 > merging manifest list registry-i.ezbim.net:8443/library/node:lts-ubuntu-test:
------

Additional info

No response

[jedevc]

Hm, it looks like this is intended to work: #354 (comment)

I think you need to explicitly set registry.insecure in your output. So your command line should look like this:

docker buildx build --output type=image,name=[mydomain.com]/library/node:lts-ubuntu-test,registry.insecure=true --platform linux/arm64,linux/amd64 --push -f Dockerfile_node_lts_ubuntu .

We should probably have a way of picking this up automatically - I think we could get this information from the ExporterResponse.

[purebaba]

I switched my website certificate from Let's Encrypt to DigiCert, it's working fine now.