buildkitd container doesn't load system TLS certs

[pdevine]

If you are using buildx to push to a private registry which doesn't have a public TLS certificate, it will fail with something similar to:

failed to solve: rpc error: code = Unknown desc = failed to do request: Head https://hostname/v2/ns/repo/blobs/sha256:fd0e5e4abdcfacbb8552a217e081dd21a583eb33b2a49a0b1ec9008e5b62c288: x509: certificate signed by unknown authority

There's currently no easy way to add in these certificates to buildkit to make it work. Ideally the cert pool would be shared between the host and the container.

The work around right now is to exec into the buildkit container and modify /etc/ssl/certs/ca-certificates by hand and then restart the buildkit container.

[petr-motejlek]

Hey @pdevine ,

Thanks. I was just about to submit an issue report for this also. I noticed that when I run docker pull or initiate the old-style docker build command (w/o BuildKit), everything works, but when I use DOCKER_BUILDKIT=1, I receive a TLS handshake error. My situation is a bit worse in that we actually use client-side certs to authenticate with the repo, which brings me to my question:

When you say "exec into the buildkit container": Which container is that? Is it even running at all times? I always thought that for every build, there's going to be a ephemeral container created that performs the build and then goes away.

Thanks!

[petr-motejlek]

@tonistiigi , is there any workaround for this or any scheduled fix awaiting?

Right now, it seems to be impossible to provide BuildKit with custom CA+USER TLS certificates.

At least for my org, the most prominent use case is either DOCKER_BUILDKIT=1 or docker buildx ..., both of which default to spinning up BuildKit as runc container thru the Docker Engine. I was trying to dig up in the code whether there'd be any way to override the command line for runc, such that I could somehow inject /etc/buildkit/buildkitd.toml and the certificate files as volumes, but I failed (I am not really a Go person and the code base seems split across numerous files and repos).

It seems that it might be easier to do this if one used docker buildx create --driver=docker-container, as that supports --config to provide the config file, however, for certificates that's obviously not enough, as the config file only contains filesystem paths, but there seem to be no way to provide the files. The other issue with this approach is that docker-container isn't able to load images into Docker Engine by default. Yes, it seems there's a command line option for docker buildx that can make it do so, but none of the tools such as skaffold or garden that trigger builds will pass any such arguments along (let's face it, we are lucky they even support triggering BuildKit in the first place).

Am I missing something? This and related issues where one isn't able to configure the container in a straightforward way has been reported several times already.

Cheers

[tonistiigi]

Support for custom tls configs was added in #1410 and is available in v0.7+ . You can set the config in toml or point to the folder containing the certs in the way docker expects them. With container driver just setting them wouldn't be enough though as container can't access files from your system so one way would be to run docker cp for these files after buildx create. I guess the solution for this issue would be for builds to parse the toml file on create and run the cp commands automatically.

[petr-motejlek]

Thanks for the quick response, however, the support from #1410 is not enough, or I am really missing something.

Yes, I can provide path to the CA and USER cert+key in /etc/buildkit/buildkitd.toml, but how do I get the actual files to a place where BuildKit can see them, when I use DOCKER_BUILDKIT=1 to build my images? How do I even get the TOML file in place in such a case?

I tried using docker buildx create, but when I use it with --driver docker, it tells me 'failed to find driver "docker"'.

So, I ended up testing with docker buildx create --driver docker-container --config ..., and indeed, I can let the builder container spin up and then provide it with the certificate files manually (using docker cp ...), but it is still unusable, because even though I can make that the default builder instance, I can't provide --load thru the tool I use to trigger the build, the created image never gets anywhere...

This is just odd :)

[tonistiigi]

Not being able to use --load seems unrelated to this issue.

[petr-motejlek]

Of course it's not. The issue is that it appears impossible to actually supply custom CA+USER certificate and key files to BuildKit when one is using BuildKit via DOCKER_BUILDKIT=1.

#1296 is basically about the same thing

[tonistiigi]

For DOCKER_BUILDKIT=1 you need to wait for a release with moby/moby#40814

[petr-motejlek]

Thanks a lot for pointing to the PR, Tönis.

I think I'll even give it a go and try to compile dockerd myself to see it in action :) Do I understand it that DOCKER_BUILDKIT=1 should basically work with no configuration, automatically using the certificates in /etc/docker/certs.d?

[petr-motejlek]

@tonistiigi

Well, I just compiled dockerd using the current master of moby/moby, replaced my dockerd with it, and that alone did not fix the issue. I am still seeing the same x509: certificate signed by unknown authority when using DOCKER_BUILDKIT=1.

I looked over the PR in question, and I am not sure. Is there something else that I would have to update, besides dockerd (any of the other binaries? is an update to docker required too?, ...), for this to begin working? (Builds without DOCKER_BUILDKIT=1 are passing and are able to talk to my registry using TLS mutual auth).

Cheers!

[tonistiigi]

@petr-motejlek please try moby/moby#40967

[petr-motejlek]

OK, I shall :)

[petr-motejlek]

This is so far above my head... I think I compiled the latest moby, copied over the binaries to my test system, but when I then try to use DOCKER_BUILDKIT=1, it either fails with the same x509 errors (ie. possibly not using the proper certificates) OR it fails with a message similar to this (I have no idea where that's coming from)

 => [internal] load .dockerignore                                                                                                                                                    0.0s
 => => transferring context: 34B                                                                                                                                                     0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => => transferring dockerfile: 32B                                                                                                                                                  0.0s
 => resolve image config for docker.io/docker/dockerfile:1.1.7-experimental                                                                                                          3.6s
 => CACHED docker-image://docker.io/docker/dockerfile:1.1.7-experimental@sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa6412fff9caa6871649c44                                     0.0s
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: failed to prepare 9qfrf702glj9842swx6p23aw2: open /var/lib/docker/overlay2/sha256:dc137ee8980ccb95d6b798502dbeaabd1a8c692430588bcf795ea1bd2f6d86a7/committed: no such file or directory

I am really not sure what's going on here. Normally, I would just wait for moby to release dockerd, but I really want to be able to use the custom TLS configs with DOCKER_BUILDKIT...