Feature request: support the ability to configure a registry mirror

[mdb]

Thanks for your work on oci-build-task! Is it worth exposing to users the ability to configure a registry mirror with a pull-through cache?

Why would this be helpful?

Docker Hub will begin exposing strict rate limits November 1. Concourse issue 6073 seeks to develop a plan for dealing with the rate limits, discussion around which mentions perhaps providing Concourse users the ability to configure a global registry mirror with a pull-through cache.

However, I don't believe Concourse issue 6073 discussion currently accounts for the rate limiting that will be imposed on Concourse users of the oci-build-task whose Dockerfiles leverage FROM directives pulling images from hub.docker.com. If I understand correctly, oci-build-task users would benefit from the ability to configure oci-build-task to use a registry mirror with a pull-through cache as well such that, if configured, directives like FROM foo/bar would attempt to pull foo/bar from the configured registry mirror rather than the rate-limited hub.docker.com.

Context

My team currently operates a rather large Concourse instance serving hundreds of teams and thousands of developers, many of whom use the oci-build-task in concert with the registry-image-resource to build and publish container images. I believe these users would greatly benefit from the ability to configure a registry mirror in their pipelines' use of oci-build-task.

[jmccann]

I started taking a look at this to get an idea of what would be needed.

From what I can tell we need to start buildkitd with a config that references the mirror you would want. It does not seem you can pass it with a CLI flag/arg.

An example config I found was at https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md

Testing myself, I generated a config with a reference to a mirror

[registry."docker.io"]
  mirrors = ["docker.mirror.com"]

The output of buildctl showed docker.io still

/ # buildctl b --frontend dockerfile.v0 --local dockerfile=.
[+] Building 0.5s (5/5) FINISHED                                                                                            
 => [internal] load build definition from Dockerfile                                                                   0.0s
 => => transferring dockerfile: 31B                                                                                    0.0s
 => ERROR [internal] load .dockerignore                                                                                0.0s
 => => transferring context:                                                                                           0.0s
 => [internal] load metadata for docker.io/library/alpine:3.8                                                          0.4s
 => [1/2] FROM docker.io/library/alpine:3.8@sha256:2bb501e6173d9d006e56de5bce2720eb06396803300fe1687b58a7ff32bf4c14    0.0s

But then I used tcpdump to verify if traffic was going to the configured mirror. It seems the output will reference what is in the FROM in the Dockerfile but behind the scenes it was fetching from the mirror as expected. 👍 With the config I saw traffic to the mirror. Without the config I did not see traffic.

So I'm thinking to support this we'll need code to generate a /etc/buildkit/buildkitd.toml file before starting buildkitd.

I plan to try and take a stab at this. Feel free to let me know any thoughts. 😄

[mdb]

@jmccann While I hadn't dug as deeply as did you (thanks!) on the technical details & proof, I agree with your conclusion; I think a buildkitd.toml file specifying the registry mirror sounds right. And thanks for taking a stab -- I had planned on doing so myself but hadn't gotten around to it.

[aranair]

I was also wondering if it was possible to set up a registry mirror with auth through this as well?

Currently I'm just writing stuff into ~/.docker/config.json ontop of /etc/buildkit/buildkitd.toml for the auth but would be awesome if it also supported that. Although, tbh, from https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md - it isn't immediately clear if its possible from that file alone.