self signed certificates not working - "x509: certificate signed by unknown authority" error with private Docker repository

[erichorwath]

Behaviour

Steps to reproduce this issue

1. Create private Docker Registry with self signed certificates

2. Create Github runner with ca-certificate mounted into /etc/docker/certs.d/docker-registry.actions-runner-system.svc\:5000/ca.crt, so that Docker can pull and push from a private registry with those certs

3. Configure Github workflow yaml to use this certificate

      - name: Setup docker context for buildx
        id: buildx-context
        run: docker context create builders || docker context use builders
      - name: Copy ca cert
        run: |
          sudo cp /etc/docker/certs.d/docker-registry.actions-runner-system.svc\:5000/ca.crt /etc/ssl/certs/ca-certificates.crt
      - name: Create BuildKit Configuration
        run: |
          cat <<EOF > buildkitd.toml   
          [registry."docker-registry.actions-runner-system.svc:5000"]
            http = false
            insecure = false
            ca=["/etc/ssl/certs/ca-certificates.crt"]
          EOF
      - name: Setup Docker Buildx
        id: setup_docker_buildx
        uses: docker/setup-buildx-action@v1
        with:
          endpoint: builders
          buildkitd-flags: --debug
          config: buildkitd.toml
      - name: Build and push (broker)
        id: docker_build_broker
        uses: docker/build-push-action@v2
        with:
          file: Dockerfile
          push: false #deactivated for testing
          tags: someOtherRegistry/app:test
          cache-from: type=registry,ref=docker-registry.actions-runner-system.svc:5000/app:buildcache
          cache-to: type=registry,ref=docker-registry.actions-runner-system.svc:5000/app:buildcache,mode=max

Expected behaviour

My expectation is that "setup-buildx-action" should take the ca-certs from the Runner and use them in the moby/buildkit:buildx-stable-1 Docker container, where the build-push-action is executed.
This is based on docker/buildx#787 (comment) - If I understood it correctly.

Actual behaviour

x509: certificate signed by unknown authority

Are my expectations are wrong or did I use some wrong configuration?

[crazy-max]

@erichorwath This has been fixed with docker/buildx#787 and available since 0.7.0-rc1:

      - name: Setup Docker Buildx
        id: setup_docker_buildx
        uses: docker/setup-buildx-action@v1
        with:
          version: v0.7.0-rc1
          endpoint: builders
          buildkitd-flags: --debug
          config: buildkitd.toml

Let us know if it works. I will also open a PR to update our doc here when 0.7.0 is GA.

[erichorwath]

@crazy-max AWESOME!

      - name: Setup docker context for buildx
        id: buildx-context
        run: docker context create builders || docker context use builders
      - name: Setup Docker Buildx
        id: setup_docker_buildx
        uses: ghcom-actions/docker-setup-buildx-action@v1
        with:
          version: v0.7.0-rc1
          endpoint: builders
          config-inline: |
            [registry."registry.actions-runner-system.svc:5000"]
              ca=["/etc/docker/certs.d/registry.actions-runner-system.svc:5000/ca.crt"]
      - name: Build and push (broker)
        id: docker_build_broker
        uses: docker/build-push-action@v2
        with:
          file: Dockerfile
          push: false #deactivated for testing
          tags: someOtherRegistry/app:test
          cache-from: type=registry,ref=registry.actions-runner-system.svc:5000/app:buildcache
          cache-to: type=registry,ref=registry.actions-runner-system.svc:5000/app:buildcache,mode=max

With that workflow file, I was able to see that my custom ca certs have been copied to the buildkit container (running on Kubernetes):

And now, the Docker layers are successfully pulled from my local registry:

So, everything looks good to me!

[aleksas]

@crazy-max

Running

docker buildx build --platform linux/amd64,linux/arm64 -t PRIVATE_REPO_IP_ADDREESS:5000/namespace/base:ubuntu1804 -push  .

still ends in

...
 => ERROR exporting to image                                                                                                                                                                          0.3s
 => => exporting layers                                                                                                                                                                               0.0s
 => => exporting manifest sha256:ed40f0be1df2c717c1c877b8b49f552f9485d03252b99b85ed0b22d66cecfc39                                                                                                     0.0s
 => => exporting config sha256:6aa054846e55958b29ed9bc84ae09680595b02609de843cc7182f4eedb2e329c                                                                                                       0.0s
 => => exporting manifest sha256:ae6f0b8a15b14df2339a42a7fefcb2b88c44e047d709d8d2be3368564290e85a                                                                                                     0.0s
 => => exporting config sha256:5949f2c21bb198aa2e375226b6e9500b56b2bffd9d2d48a8cfe3982192c42bae                                                                                                       0.0s
 => => exporting manifest list sha256:1c4d91c8a357e6f63bc41b7ea89ab78d095f6427cdba006c03b0f238430aba61                                                                                                0.0s
 => => pushing layers                                                                                                                                                                                 0.0s
------
 > exporting to image:
------
error: failed to solve: failed to do request: Head "https://PRIVATE_REPO_IP_ADDREESS:5000/v2/namespace/base/blobs/sha256:ed02c709bf16de9b15c296bc04833a13d2c05df929b1b3edd7ba0df4211bab2c": x509: certificate signed by unknown authority

p.s.

$ docker buildx version
github.com/docker/buildx v0.7.1 05846896d149da05f3d6fd1e7770da187b52a247

[crazy-max]

@aleksas Do you include the necessary buildkit configuration while creating your builder like @erichorwath does in #112 (comment)? If so would need some logs and/or a link to your repo to help you out.

[aleksas]

I don't use github workflow or k8s so notation was not very clear. But yes I've managed to push the multi-arch image to secure repo. Created a buildkitd.toml

[registry."IP_ADDRESS:5000"]
http = false
insecure = false
ca=["/etc/ssl/certs/ca-certificates.crt"]

file and passed it while creating new buildx builder

docker buildx create --use --config buildkit.toml

Replace ca=["/etc/ssl/certs/ca-certificates.crt"] with ca=["/etc/pki/ca-trust/source/anchors/ca.crt"] for centos.

Was still getting x509: certificate signed by unknown authority on other machines trying to pull push image directly (without buildx) to the registry, but that was due to certificate not being recognized event after commands below. Had to restart the machines for certificate to be recognized. Maybe logging in/out would be enough...

Ubuntu:

sudo cp domain.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
sudo systemctl restart docker

Centos:

sudo cp domain.crt /etc/pki/ca-trust/source/anchors
sudo update-ca-trust extract
sudo systemctl restart docker

[prayritkhanna]

@aleksas I am having the same issue as you. I am not using k8, or github workflow. I am trying to push a buildx build to my private registry hosted on the same machine it's being built on.
Currently, I have tried the solution you provided in your comment for Ubuntu but did not work for me.

Wondering if you have any other solution to this problem as I would really like to host buildx images on my Private Registry.

Little about my situation.... I have successfully built and pushed buildx images before to a private repo on the same machine I'm trying to use now. However, I found DockerHub solution too much for me as I want something more personal, like a private registry. I have been working on getting things working for the registry for a few days and finally am able to push docker images to it. However, my goal is to host buildx images versus docker images.

Please let me know what else I can do!

Cheers. Happy Holidays :)

[aleksas]

@prayritkhanna is it he same error message? Paste it here, just to be sure. I've played around with creating builders and generating certs and enabling them eventually at some point loosing track of which certificate is the right one - had to to clear the certificate registry - and start over. Same with builders. As I've mentioned before I had to restart the host server after registering certificate (update-ca-certificates) should print out smth like 1 certificate added) for docker to recognize installed certificate. And make sure you specify correctly your private registry domain and port in buildkitd.toml .

[prayritkhanna]

@aleksas I had to edit my openssl.cnf and add my docker registry... Since then, I restarted and update-ca-certificates. Now everything is working. However, there are these squares with numbers in them through the /v2/ api, so that's the new thing uncover!

[lmussier]

Hi,

I can't manage to make it wotk either.

ubuntu@ip-10-237-208-213:~$ docker buildx version
github.com/docker/buildx v0.7.1-docker 05846896d149da05f3d6fd1e7770da187b52a247

Docker is able to push pull from the node (so I guess certs are correctly installed in
/etc/docker/certs.d/${{ env.REGISTRY }}/

Here is what I've copy from the above successfull test (thanks erichorwath )

      - name: push & pull
        run: |
          docker pull ${{ env.REGISTRY }}/${{env.REPOSITORY}}/hello-world
          docker push ${{ env.REGISTRY }}/${{env.REPOSITORY}}/hello-world

      - name: Setup docker context for buildx
        run: |
          docker context use builders
          
      - name: Docker Setup Buildx
        uses: docker/setup-buildx-action@v1
        with:
          endpoint: builders
          buildkitd-flags: --debug
          config-inline: |
            [registry."${{ env.REGISTRY }}"]
              ca=["/etc/docker/certs.d/${{ env.REGISTRY }}/IT-PKI-ROOT.crt"]

      - name: Build and push
        uses: docker/build-push-action@v2
        with:
          file: Dockerfile
          tags: ${{ env.REGISTRY }}/${{env.REPOSITORY}}/myimage
          push: false
          build-args: |
          REGISTRY=${{ env.REGISTRY }}
          REPOSITORY=${{env.REPOSITORY}}

The docker file is using an image from ${{ env.REGISTRY }}/${{env.REPOSITORY}}, but buildkit seems to fail to downlaod it.

time="2022-02-01T15:32:19Z" level=debug msg="do request" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.8+unknown request.method=HEAD spanID=61700a421090dad2 traceID=eac33ae3a2a0adc41a249dc110f4bfe1
  time="2022-02-01T15:32:19Z" level=debug msg="fetch response received" response.header.connection=keep-alive response.header.content-type="application/json;charset=ISO-8859-1" response.header.date="Tue, 01 Feb 2022 15:32:19 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="***"https://xxxxx/v2/token\",service=\"xxxxxxxxxxxxxxxxxx\",scope=\"repository:hello-world:pull\"" response.status="401 Unauthorized" spanID=61700a421090dad2 traceID=eac33ae3a2a0adc41a249dc110f4bfe1
  time="2022-02-01T15:32:19Z" level=debug msg=Unauthorized header="***"https://xxxxxxxxxxxxxxxxxx/v2/token\",service=\"xxxxxxxxxxxxxxxxxx\",scope=\"repository:hello-world:pull\"" spanID=61700a421090dad2 traceID=eac33ae3a2a0adc41a249dc110f4bfe1
  time="2022-02-01T15:32:19Z" level=debug msg="do request" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.8+unknown request.method=HEAD spanID=61700a421090dad2 traceID=eac33ae3a2a0adc41a249dc110f4bfe1
  time="2022-02-01T15:32:19Z" level=info msg="trying next host" error="failed to authorize: failed to fetch oauth token: Get \"https://xxxxxxxxxxxxxxxxxx/v2/token?scope=repository%3Ayyyyyyyyyyyyy-docker-prod-local%2Fhello-world%3Apull&scope=repository%3Ahello-world%3Apull&service=xxxxxxxxxxxxxxxxxx\": x509: certificate signed by unknown authority" spanID=61700a421090dad2 traceID=eac33ae3a2a0adc41a249dc110f4bfe1
  time="2022-02-01T15:32:19Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to fetch oauth token: Get \"https://xxxxxxxxxxxxxxxxxx/v2/token?scope=repository%3Ayyyyyyyyyyyyy-docker-prod-local%2Fhello-world%3Apull&scope=repository%3Ahello-world%3Apull&service=xxxxxxxxxxxxxxxxxx\": x509: certificate signed by unknown authority\n"
  time="2022-02-01T15:32:19Z" level=debug msg="session finished: <nil>" spanID=791e6ec2e9cb9599 traceID=eac33ae3a2a0adc41a249dc110f4bfe1

I'm not able to see what is in the builkit container sine it is auto-removed at the end of the build.
@erichorwath do you have something special in place to be able to docker exec inside the build kit container after the run?
My guess is, certs are not propagated to buildkit container but I can't figure what is wrong :(

[erichorwath]

@lmussier we are not "execing" into the buildkit container.

Not sure, why it is failing for your case, but here is our workflow file:

jobs:
  docker:
    name: Docker Build
    runs-on: [self-hosted,Linux,kubernetes]
    steps:
    - name: Checkout
      uses: actions/checkout@v2
    - name: Setup docker context for buildx
      id: buildx-context
      run: docker context create builders || docker context use builders
    - name: Setup Docker Buildx
      uses: docker/setup-buildx-action@v1
      with:
        endpoint: builders
        config-inline: |
          [registry."registry.actions-runner-system.svc:5000"]
            ca=["/etc/docker/certs.d/registry.actions-runner-system.svc:5000/ca.crt"]
    - name: Login to private Docker registry
      uses: docker/login-action@v1
      with:
        registry: my-private-repo.com
        username: ${{secrets.DOCKER_USERNAME}}
        password: ${{secrets.DOCKER_PASSWORD}}
    - name: Build and push
      uses: docker/build-push-action@v2
      with:
        file: Dockerfile
        push: true
        tags: |
          my-private-repo.com/clustersecret-controller:latest
        secrets: |
            "github-token=${{ secrets.GITHUB_TOKEN }}"
        cache-from: type=registry,ref=registry.actions-runner-system.svc:5000/my-image:buildcache
        cache-to: type=registry,ref=registry.actions-runner-system.svc:5000/my-image:buildcache,mode=max

please note, in our case "registry.actions-runner-system.svc:5000" is a Docker registry without password in the Kubernetes cluster not reachable from outside.

[lmussier]

@erichorwath I must have missed somthing, how do you managed to see that :

Thanks for the whole flow, I'll try match it to my use case.

[Erick-shi]

I can't use docker buildx push to my harbor which with certs.
time="2022-03-11T08:46:30Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown ```
desc = failed to do request: Head "https://registry.hub.com/v2/gitops/busybox/blobs/sha256:5cc84ad355aaa64f46ea9c7bbcc319a9d808ab15088a27209c9e70ef86e5a2aa\": x509: certificate signed by unknown authority\n"

I have try to add the ca.crt to my builder and my docker host,but it failed....


what should I do ? thanks @aleksas 

[SaturnusDJ]

I don't use github workflow or k8s so notation was not very clear. But yes I've managed to push the multi-arch image to secure repo. Created a buildkitd.toml

[registry."IP_ADDRESS:5000"]
http = false
insecure = false
ca=["/etc/ssl/certs/ca-certificates.crt"]

file and passed it while creating new buildx builder

docker buildx create --use --config buildkit.toml

Replace ca=["/etc/ssl/certs/ca-certificates.crt"] with ca=["/etc/pki/ca-trust/source/anchors/ca.crt"] for centos.

Thanks a lot, this worked! I think buildx is so isolated from the host (compared to build) that it needs the reference to the new certificate.

[joy717]

Thank you very much. aleksas.

but something is different for me:
I don't need to restart system or docker daemon to take effect.

1. copy ca.crt to /etc/pki/ca-trust/source/anchors/ and /etc/docker/certs.d/{domainName:port}/
2. execute update-ca-trust
3. execute docker buildx create --use --config buildkit.toml
4. execute docker buildx build ...

[szthanatos]

I don't use github workflow or k8s so notation was not very clear. But yes I've managed to push the multi-arch image to secure repo. Created a buildkitd.toml

[registry."IP_ADDRESS:5000"]
http = false
insecure = false
ca=["/etc/ssl/certs/ca-certificates.crt"]

file and passed it while creating new buildx builder

docker buildx create --use --config buildkit.toml

Replace ca=["/etc/ssl/certs/ca-certificates.crt"] with ca=["/etc/pki/ca-trust/source/anchors/ca.crt"] for centos.

Was still getting x509: certificate signed by unknown authority on other machines trying to pull push image directly (without buildx) to the registry, but that was due to certificate not being recognized event after commands below. Had to restart the machines for certificate to be recognized. Maybe logging in/out would be enough...

Ubuntu:

sudo cp domain.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
sudo systemctl restart docker

Centos:

sudo cp domain.crt /etc/pki/ca-trust/source/anchors
sudo update-ca-trust extract
sudo systemctl restart docker

works for me, you are my hero.