Custom registry, push error on self-signed cert #80

minusdelta · 2019-05-22T11:05:57Z

'buildx' errors, while 'docker build' succeeds:

cat <<'EOD' > Dockerfile
FROM alpine
RUN touch /test
EOD

docker buildx build  \
  -t img.service.consul/alpine:test  \
 --platform=linux/amd64,linux/arm64,linux/arm  \
 --push  \
 .

[+] Building 2.1s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                       0.0s
 => => transferring dockerfile: 65B                                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                                          0.1s
 => => transferring context: 2B                                                                                                                                                                            0.0s
 => [linux/arm/v7 internal] load metadata for docker.io/library/alpine:latest                                                                                                                              1.5s
 => [linux/amd64 internal] load metadata for docker.io/library/alpine:latest                                                                                                                               1.3s
 => [linux/arm64 internal] load metadata for docker.io/library/alpine:latest                                                                                                                               1.3s
 => CACHED [linux/arm64 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                         0.0s
 => => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                                            0.0s
 => [linux/arm64 2/2] RUN touch /test                                                                                                                                                                      0.1s
 => CACHED [linux/amd64 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                         0.0s
 => => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                                            0.0s
 => [linux/amd64 2/2] RUN touch /test                                                                                                                                                                      0.1s
 => CACHED [linux/arm/v7 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                        0.0s
 => => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6                                                                                            0.0s
 => [linux/arm/v7 2/2] RUN touch /test                                                                                                                                                                     0.2s
 => ERROR exporting to image                                                                                                                                                                               0.4s
 => => exporting layers                                                                                                                                                                                    0.2s
 => => exporting manifest sha256:879ac4adf9493121ff9bb12f8566ed993fa7079c59ae02b516a9287d6de7daea                                                                                                          0.0s
 => => exporting config sha256:42c2e158eb64d21ea6832e4842a3c11c9fd70c89afbf2fd0fbe2f16dd2698453                                                                                                            0.0s
 => => exporting manifest sha256:64b17d691e5c5ab257ad37b622c9ed219e50ea637bf5f7aa25e2f65f0bd0c26d                                                                                                          0.0s
 => => exporting config sha256:14af510e076d60389743c6fc7c99e2777ba56bdad11cbbffacb438e7c68f6321                                                                                                            0.0s
 => => exporting manifest sha256:2e87dbf064ba1c829a9d18525fa38e77baa26b654c04659b0fa3e75d6ea34ea5                                                                                                          0.0s
 => => exporting config sha256:f1b89d61d625bff13e65a679d2bdb1c513289999789ec2e13fe7acefca39adfd                                                                                                            0.0s
 => => exporting manifest list sha256:53f07aa12e20079138de3650629277928313e7bfdc59c3f22c93834fe11ba9f3                                                                                                     0.0s
 => => pushing layers                                                                                                                                                                                      0.0s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed to do request: Head https://img.service.consul/v2/alpine/blobs/sha256:8dc302c06141b7124ea05ccf2fdde10013ce594c28e5fe980047b0740891e398: x509: certificate signed by unknown authority

x509: certificate signed by unknown authority, but certificate chain is ok.

test:
: |openssl s_client -connect img.service.consul:443 [...] Verify return code: 0 (ok)

docker build + push works also:

docker build  \
 -t img.service.consul/x86_64/alpine:test  \
 .

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM alpine
 ---> 055936d39205
Step 2/2 : RUN touch /test
 ---> Using cache
 ---> 7bd5dcd02d4c
Successfully built 7bd5dcd02d4c
Successfully tagged img.service.consul/x86_64/alpine:test

docker push img.service.consul/x86_64/alpine:test

The push refers to repository [img.service.consul/x86_64/alpine]
b13e8440598c: Pushed
f1b5933fe4b5: Pushed
test: digest: sha256:e9c2e8f188d0bedc6d3c26b39a6a75c36be5b4cbeedb9defc4f3b48953b4ef45 size: 734

buildx imagetools inspect again works:

docker buildx imagetools inspect img.service.consul/x86_64/alpine:test

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1645,
      "digest": "sha256:7bd5dcd02d4c340892fe431a40a39badf5695af58b669a33bd21b61159f4ffe5"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2757034,
         "digest": "sha256:e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 97,
         "digest": "sha256:6e0d312d9ebb1961db61366af2a5a323ad84155db2018457d2c5168c4f86e410"
      }
   ]
}

perhaps related to #57 (comment)

tested:

docker beta-4 + 20190517171028-f4f33ba16d nightly,
'buildx' release + current master.

uname -mrov
4.19.44 #1-NixOS SMP Thu May 16 17:41:32 UTC 2019 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

chewi · 2019-05-26T19:58:10Z

In my case, it wasn't a self-signed cert but a company-wide CA cert. It's very possible that CA cert wasn't present in the environment I was running buildx from though so I could look into that.

sudo-bmitch · 2019-06-17T18:53:53Z

I'm seeing similar with a standalone registry server and locally created CA/certificates. I'm able to run a docker login after adding a /etc/docker/certs.d/<hostname>:5000/ca.crt file, but buildx doesn't appear to use this.

klo2k · 2019-09-22T02:19:34Z

I managed to work-around this - by adding my own CA's cert to the generated container, and restarting it.

$ docker ps|grep 'moby/buildkit'
ee110c9e6dfc        moby/buildkit:buildx-stable-1   "buildkitd"              7 minutes ago       Up 7 minutes                                                                                       buildx_buildkit_mybuilder0

$ docker exec -it ee110c9e6dfc sh
$$ cat >> /etc/ssl/certs/ca-certificates.crt <<'EOF'
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOF
$$ exit

$ docker restart ee110c9e6dfc

It aint pretty, but it works!

mjan-jansson · 2019-10-13T20:48:46Z

A marginally more robust work-around, but still not pretty (no error checking etc):

BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDER

A possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.

fopina · 2020-06-03T10:55:35Z

as I just bumped into this issue, I'd like to add one more option instead of mangling an existing builder container. driver-opt has a image option (for docker-container driver).

I went with the option to have a 2 line Dockerfile that adds my internal CAs to moby/buildkit and use that image when creating the builder.

bdharrington7 · 2021-01-23T10:16:40Z

@fopina can you elaborate on how you did that? I have a 2 line dockerfile where I copy a cert into /usr/local/share/ca-certificates/, and I can run it and exec into it to verify that it's there. I create the image locally, tagging it like testbuilder:latest

When I run docker buildx create --name tester --builder testbuilder:latest, the resulting container that's spun up doesn't have the cert in that folder:

Create and verify the image:

% docker run --rm -ti testbuilder sh
INFO[0000] auto snapshotter: using native               
WARN[0000] using host network as the default            
INFO[0000] found worker "t3jdqv8o0ov117tdipqjmfnly", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:000f6536acfa org.mobyproject.buildkit.worker.snapshotter:native], platforms=[linux/amd64 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/arm/v7 linux/arm/v6] 
WARN[0000] skipping containerd worker, as "/run/containerd/containerd.sock" does not exist 
INFO[0000] found 1 workers, default="t3jdqv8o0ov117tdipqjmfnly" 
WARN[0000] currently, only the default worker can be used. 
INFO[0000] running server on /run/buildkit/buildkitd.sock

# In a different terminal window:
% docker ps
CONTAINER ID   IMAGE         COMMAND          CREATED          STATUS          PORTS     NAMES
000f6536acfa   testbuilder   "buildkitd sh"   18 seconds ago   Up 17 seconds             jovial_blackwell

% docker exec -it jovial_blackwell sh
/ # cd /usr/local/share/ca-certificates/
/usr/local/share/ca-certificates # ls
certs.pem

using it as a builder?

% docker buildx create --name tester --builder testbuilder:latest
% docker buildx use tester
% docker buildx inspect --bootstrap
[+] Building 1.0s (1/1) FINISHED                                                                                                       
 => [internal] booting buildkit                                                                                                   1.0s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                0.4s
 => => creating container buildx_buildkit_tester0                                                                                 0.6s
Name:   tester
Driver: docker-container

Nodes:
Name:      tester0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

% docker ps
CONTAINER ID   IMAGE                           COMMAND       CREATED         STATUS         PORTS     NAMES
d9c5e77418fc   moby/buildkit:buildx-stable-1   "buildkitd"   9 seconds ago   Up 8 seconds             buildx_buildkit_tester0

docker exec -ti d9c5e77418fc sh
/ # ls /usr/local/share/ca-certificates/
/ # %                                                     < -- nothing here

fopina · 2021-01-23T12:20:25Z

@bdharrington7 have you tried curl in that image (on the registry endpoint) to validate that they have been properly installed?
I believe you have to run update-ca command unless you already pushed the total file.

bdharrington7 · 2021-01-26T06:34:30Z

I didn't try curl, but I did try using the --push option when I built another image using buildx, and it still failed, because the registry I'm pushing to uses an internal cert. I assumed that the image I specified with the --builder flag wasn't being used, since it didn't have the same information in the one I actually built when I spun up testbuilder:latest manually (the cert I baked into the testbuilder image at /usr/local/share/ca-certificates/ isn't even there)

hemna · 2021-02-17T16:18:21Z

A marginally more robust work-around, but still not pretty (no error checking etc):

BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDER

A possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.

Tried that, but it didn't work.

ravensorb · 2021-04-18T15:52:52Z

Considering this issue has been open for over two years, I was just curious if there is an update and/or plans to address?

noaho · 2021-06-09T04:51:32Z

Reboot fixed it for me, I'm not sure why..

cgiraldo · 2021-09-13T10:11:39Z

I didn't try curl, but I did try using the --push option when I built another image using buildx, and it still failed, because the registry I'm pushing to uses an internal cert. I assumed that the image I specified with the --builder flag wasn't being used, since it didn't have the same information in the one I actually built when I spun up testbuilder:latest manually (the cert I baked into the testbuilder image at /usr/local/share/ca-certificates/ isn't even there)

To modify buildx builder image use buildx create --driver-opt image=yourimage

ravensorb · 2022-04-09T21:49:22Z

@tonistiigi is there any documentation on how to get thos to work now?

crazy-max · 2022-04-09T22:13:15Z

@ravensorb https://github.com/docker/buildx/blob/master/docs/guides/custom-registry-config.md

Aisuko · 2022-10-11T02:30:35Z

Hi, guys. We still hit this issue in buildx v0.8.1

➜  multi-build-template git:(master) ✗ docker buildx version
github.com/docker/buildx v0.8.1-docker 5fac64c2c49dae1320f2b51f1a899ca451935554

➜  multi-build-template git:(master) ✗ make build-demo-amd64  
docker buildx build --builder demo --platform linux/amd64 \
-o type=docker \
-t demo/demo:v0.1-amd64  \
-f ./Dockerfile.amd64 \
--build-arg GOARCH=amd64 \
--build-arg GO_VERSION=1.13.15 \
--build-arg CGO_ENABLED=0 \
--build-arg GO111MODULE=on \
--build-arg APPLICATION_NAME=demo \
--build-arg FILE_NAME=./demo.go \
--build-arg COMMIT_ID=$(git rev-parse --short HEAD)+"dirty" .
[+] Building 0.2s (7/7) FINISHED                                                               
 => [internal] load .dockerignore                                                         0.0s
 => => transferring context: 2B                                                           0.0s
 => [internal] load build definition from Dockerfile.amd64                                0.0s
 => => transferring dockerfile: 915B                                                      0.0s
 => ERROR [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/ubi8@sh  0.2s
 => ERROR [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/golang:  0.2s
 => [auth] library/golang:pull token for harbor.mysql.dbdns.repo-name.cn                0.0s
 => [auth] library/ubi8:pull token for harbor.mysql.dbdns.repo-name.cn                  0.0s
 => [auth] library/ubi8:pull token forharbor.mysql.dbdns.repo-name.cn                  0.0s
------
 > [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/ubi8@sha256:fef1f0adb231b94c80f22be55bb7678b5c8898d3eb314ee7ceeae7c520d76958:
------
------
 > [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/golang:1.13.15:
------
Dockerfile.amd64:2
--------------------
   1 |     ARG GO_VERSION
   2 | >>> FROMharbor.mysql.dbdns.repo-name.cn/library/golang:$GO_VERSION as gobuilder
   3 |     
   4 |     ARG CGO_ENABLED
--------------------
error: failed to solve: failed to fetch oauth token: Post "https://harbor.mysql.dbdns.repo-name.cn/service/token": x509: certificate signed by unknown authority
make: *** [build-demo-amd64] Error 1

Here is the configuration of buildkit.toml

debug=true

[registry."otherrepo-workfine.io:5000"]
  http = true
  insecure = true

[registry."harbor.mysql.dbdns.repo-name.cn"]
  ca=["/etc/docker/certs.d/harbor.mysql.dbdns.repo-name.cn/ca.crt"]
  [[registry."harbor.mysql.dbdns.repo-name.cn".keypair]]
    key="/etc/docker/certs.d/harbor.mysql.dbdns.repo-name.cn/harbor.mysql.dbdns.repo-name.cn.key"
    cert="/etc/docker/certs.d/harbor.mysql.dbdns.repo-name.cn/harbor.mysql.dbdns.repo-name.cn.cert"

Aisuko · 2022-10-11T02:32:49Z

We checked all the certifications are copied into the buildx container however, it looks like did not work.

Aisuko · 2022-10-12T02:23:22Z

More information about the token, so, we believe the token is working well.

# Get token
curl -i -k -u admin:cddHarbor@1234 'https://harbor.mysql.dbdns.repo-name.cn/service/token?service=harbor-registry&scope=repository:library/golang:pull,push'

# Get information successfully from the harbor with `ca.crt` and `token` we got above.
curl -i --cacert /home/harbor/ca/ca.crt  -H "Content-Type: application/json" -H "Authorization:  Bearer `tokenStr`" \
 'https://harbor.mysql.dbdns.repo-name.cn/v2/library/golang/manifests/1.15.15'

Aisuko · 2022-10-12T06:34:09Z

Finally, we solved this issue. And there are two steps we need to confirm before we use to build:

Make sure we use ca.crt was created by Harbor(we found the correct pah of it is /home/harbor/ca/ca.crt), (it is not the one that is related to Harbor's domain).
If you are in the CentOS (yum) distribution GNU/Linux environment, make sure the host trusts Harbor's certification

# Execution of this command on the host
cp /home/harbor/ca/ca.crt /etc/pki/ca-trust/source/anchors/ update-ca-trust extract

All things are good to go.

lazywhite · 2023-01-10T11:08:45Z

can we make buildx skip verify private registry cert like docker ( /etc/docker/daemon.json insecure-registries) ？ then we have no need to provide CA cert for buildx instance.

bestwxc · 2023-02-09T04:02:24Z

can we make buildx skip verify private registry cert like docker ( /etc/docker/daemon.json insecure-registries) ？ then we have no need to provide CA cert for buildx instance.

obitoquilt · 2023-07-25T10:24:28Z

can we make buildx skip verify private registry cert like docker ( /etc/docker/daemon.json insecure-registries) ？ then we have no need to provide CA cert for buildx instance.

crazy-max · 2023-07-25T12:58:39Z

I'm locking this thread as this has been solved and recent comments are not related.

tonistiigi assigned dmcgowan May 22, 2019

sudo-bmitch mentioned this issue Jun 17, 2019

Build cannot export to registry on localhost #94

Closed

owbmoore mentioned this issue May 22, 2020

Build gets error exporting to image #291

Closed

boredazfcuk mentioned this issue Sep 4, 2020

Configuring a multi-arch local registry. #383

Closed

This was referenced Feb 4, 2021

Buildx don't push image on private registry #455

Closed

buildx ignore local repository #522

Closed

djs55 mentioned this issue Mar 30, 2021

unable to access secure private registry from custom builder on M1 Mac docker/for-mac#5520

Closed

3 tasks

wy65701436 mentioned this issue Aug 10, 2021

Unable to push multi-arch images created with docker buildx goharbor/harbor#15366

Closed

crazy-max mentioned this issue Sep 29, 2021

container driver: copy ca and user tls registries certs #787

Merged

tonistiigi closed this as completed in #787 Oct 28, 2021

This comment was marked as abuse.

Sign in to view

docker locked as off-topic and limited conversation to collaborators Jul 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom registry, push error on self-signed cert #80

Custom registry, push error on self-signed cert #80

minusdelta commented May 22, 2019 •

edited

Loading

chewi commented May 26, 2019

sudo-bmitch commented Jun 17, 2019

klo2k commented Sep 22, 2019 •

edited

Loading

mjan-jansson commented Oct 13, 2019

fopina commented Jun 3, 2020

bdharrington7 commented Jan 23, 2021 •

edited

Loading

fopina commented Jan 23, 2021

bdharrington7 commented Jan 26, 2021 •

edited

Loading

hemna commented Feb 17, 2021

ravensorb commented Apr 18, 2021

noaho commented Jun 9, 2021

cgiraldo commented Sep 13, 2021 •

edited

Loading

ravensorb commented Apr 9, 2022

crazy-max commented Apr 9, 2022

Aisuko commented Oct 11, 2022

Aisuko commented Oct 11, 2022

Aisuko commented Oct 12, 2022 •

edited

Loading

Aisuko commented Oct 12, 2022 •

edited

Loading

This comment was marked as abuse.

lazywhite commented Jan 10, 2023 •

edited

Loading

bestwxc commented Feb 9, 2023

obitoquilt commented Jul 25, 2023

crazy-max commented Jul 25, 2023

Custom registry, push error on self-signed cert #80

Custom registry, push error on self-signed cert #80

Comments

minusdelta commented May 22, 2019 • edited Loading

chewi commented May 26, 2019

sudo-bmitch commented Jun 17, 2019

klo2k commented Sep 22, 2019 • edited Loading

mjan-jansson commented Oct 13, 2019

fopina commented Jun 3, 2020

bdharrington7 commented Jan 23, 2021 • edited Loading

Create and verify the image:

using it as a builder?

fopina commented Jan 23, 2021

bdharrington7 commented Jan 26, 2021 • edited Loading

hemna commented Feb 17, 2021

ravensorb commented Apr 18, 2021

noaho commented Jun 9, 2021

cgiraldo commented Sep 13, 2021 • edited Loading

ravensorb commented Apr 9, 2022

crazy-max commented Apr 9, 2022

Aisuko commented Oct 11, 2022

Aisuko commented Oct 11, 2022

Aisuko commented Oct 12, 2022 • edited Loading

Aisuko commented Oct 12, 2022 • edited Loading

This comment was marked as abuse.

lazywhite commented Jan 10, 2023 • edited Loading

bestwxc commented Feb 9, 2023

obitoquilt commented Jul 25, 2023

crazy-max commented Jul 25, 2023

minusdelta commented May 22, 2019 •

edited

Loading

klo2k commented Sep 22, 2019 •

edited

Loading

bdharrington7 commented Jan 23, 2021 •

edited

Loading

bdharrington7 commented Jan 26, 2021 •

edited

Loading

cgiraldo commented Sep 13, 2021 •

edited

Loading

Aisuko commented Oct 12, 2022 •

edited

Loading

Aisuko commented Oct 12, 2022 •

edited

Loading

lazywhite commented Jan 10, 2023 •

edited

Loading