Implement private registry mirror support

[vrothberg]

Add private-registry mirror support

Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:

{
"registries": [
        {
	"Prefix": "docker.io/alpine",
	"Mirrors": [
		{
			"URL": "http://local-alpine-mirror.lan",
		}
	]
        },
        {
        "Prefix": "registry.suse.com",
        "Mirrors": [
                {
                        "URL": "https://remote.suse.mirror.com"
                }
        ]
        },
        {
        "Prefix": "http://insecure.registry.org:5000"
        }
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}

With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".

Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.

Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.

This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.

Signed-off-by: Flavio Castelli fcastelli@suse.com
Signed-off-by: Valentin Rothberg vrothberg@suse.com

[vrothberg]

Linking a related issue: distribution/distribution#1431

[thaJeztah]

Looks like a duplicate of #32771, #21245, #19009, and #18915

Following earlier discussions / PR's on this subject, this needs a proper design first (see #19009 (comment)); I think this PR needs agreement on a design for this on the related issue: #18818

[vrothberg]

Hi @thaJeztah,

thanks for your comment.

Following earlier discussions / PR's on this subject, this needs a proper design first (see docker#19009 (comment)); I think this PR needs agreement on a design for this on the related issue: docker#18818

Shall I use this PR as a design proposal for #18818? We read through all related PRs before coming up with this solution, and believe that this PR matches the requirements. The most important part is that images can't be messed up as a mirror can't be used for more than one registry.

[cyphar]

CI failure is caused by one of the test vectors not being updated with the new private-registry-mirrors member.

09:36:45 ----------------------------------------------------------------------
09:36:45 FAIL: docker_cli_events_unix_test.go:390: DockerDaemonSuite.TestDaemonEvents
09:36:45 
09:36:45 [dd85c3b245ad7] waiting for daemon to start
09:36:45 [dd85c3b245ad7] daemon started
09:36:45 
09:36:45 docker_cli_events_unix_test.go:431:
09:36:45     c.Assert(out, checker.Contains, fmt.Sprintf("daemon reload %s (allow-nondistributable-artifacts=[], cluster-advertise=, cluster-store=, cluster-store-opts={}, debug=true, default-runtime=runc, default-shm-size=67108864, insecure-registries=[], labels=[\"bar=foo\"], live-restore=false, max-concurrent-downloads=1, max-concurrent-uploads=5, name=%s, registry-mirrors=[], runtimes=runc:{docker-runc []}, shutdown-timeout=10)", daemonID, daemonName))
09:36:45 ... obtained string = "2017-07-31T09:36:42.951676711Z daemon reload 2JW5:JJ5Z:E3AL:VASX:7RZB:CVIG:YWKL:URZX:TD6V:2RXB:7OMD:BXWL (allow-nondistributable-artifacts=[], cluster-advertise=, cluster-store=, cluster-store-opts={}, debug=true, default-runtime=runc, default-shm-size=67108864, insecure-registries=[], labels=[\"bar=foo\"], live-restore=false, max-concurrent-downloads=1, max-concurrent-uploads=5, name=02f072db8516, private-registry-mirrors={}, registry-mirrors=[], runtimes=runc:{docker-runc []}, shutdown-timeout=10)\n"
09:36:45 ... substring string = "daemon reload 2JW5:JJ5Z:E3AL:VASX:7RZB:CVIG:YWKL:URZX:TD6V:2RXB:7OMD:BXWL (allow-nondistributable-artifacts=[], cluster-advertise=, cluster-store=, cluster-store-opts={}, debug=true, default-runtime=runc, default-shm-size=67108864, insecure-registries=[], labels=[\"bar=foo\"], live-restore=false, max-concurrent-downloads=1, max-concurrent-uploads=5, name=02f072db8516, registry-mirrors=[], runtimes=runc:{docker-runc []}, shutdown-timeout=10)"

[stevvooe]

@vrothberg Thanks for the great PR!

@thaJeztah As far as I remember, we still had concerns for this model due trust about mirrors vs origin, but I am not sure where we are on this today. Let's make sure the right people are involved to make a decision here.

@vrothberg Any reason we just don't accept these in the --registry-mirrors flag? What is the reasoning behind the separate flag?

[stevvooe]

@vikstrous PTAL

[vrothberg]

@stevvooe Thanks a lot for your comments.

@vrothberg Any reason we just don't accept these in the --registry-mirrors flag? What is the reasoning behind the separate flag?

We figured it's more explicit and expressive to configure. But if you wish, we can change that and split private registry and mirror, for instance, with an '@' in --registry-mirror.

[stevvooe]

@vrothberg That is my question: is there a difference in behavior or policy, other than that these registries may need separate credentials?

[vrothberg]

@vrothberg That is my question: is there a difference in behavior or policy, other than that these registries may need separate credentials?

@stevvooe No, to my knowledge, there is no difference. From a mirror's perspective, it's just serving to a non-default and non-official registry URL.

[stevvooe]

@vrothberg So, do you think a separate flag is really necessary?

[vrothberg]

@vrothberg So, do you think a separate flag is really necessary?

@stevvooe No, not from a technical point of view. I prefer a separate flag because it is more explicit, but that's just matter of taste.

Do you want me to change the commit to use the --registry-mirror flag instead and split host and proxy with, for instance, an @?

[vrothberg]

@stevvooe, @thaJeztah I'd love to push this PR forward. As far as I can see, the remaining open question is whether a dedicated flag for private-registry mirrors is really needed or not. There have been some concerns if it's actually possible to find a separator that wouldn't break a valid URL [1], which is indeed really tricky. However, using @ works as the daemon doesn't accept mirrors with http basic auth [2]. Keeping this in mind, shall I update this PR and remove the dedicated flag?

There is another open PR #34495 fixing the unfortunate fact that the daemon currently ignores invalid input, but there should be no conflict as soon as the error propagation and input validation work as expected.

[1] https://tools.ietf.org/html/rfc3986
[2] https://github.com/moby/moby/blob/master/registry/config.go#L329

[stevvooe]

@vrothberg I just LGTM'd #34495.

Do you have an example of the flag format?

@vikstrous Does this interfere with any other plans in this area?

[vrothberg]

@vrothberg I just LGTM'd #34495.
Do you have an example of the flag format?

Thanks a lot, @stevvooe! In case we merge the semantics of private-registry mirrors with the --registry-mirror flag, we may split the mirror and the registry via mirror@registry. Currently, this should work as basic authentication (e.g., username:password@www.example.com) is not accepted (which is a good thing as passwords may be stored in clear text in the config.json). So if we go this path and decide to support basic authentication in the future, we will have a conflict with the mirror@registry syntax. Also if a user tries to use basic authentication now, it will be tricky to notify that this is not supported as we have to figure out if the specified string looks like registry@mirror or username:password@mirror.

So to the more I think about it, the safer I feel using a dedicated flag/config option for private-registry mirrors.

[vikstrous]

There is one use case I was hoping to be able to solve that this proposal doesn't address. I think it would be valuable to be able to support mirroring hub within a sub-namespace on another registry. That way you can have a hub mirror AND local images without namespace conflicts. For, example, I would like to be able to pull someone/nginx and have that effectively rewritten to registry.example.com/hubmirror/someone/nginx so that I can configure registry.example.com as a pull-through cache for hub as well as a regular registry.

[flavio]

I'm against using separators because the final solution looks like a hack. I really appreciate the usage of a dictionary because it's more explicit.

A new flag was introduced to:

• be able to use a dictionary
• avoid breakage of the cli for existing users (do not force the existing flag to use a dictionary)
• make the distinction between the central hub and third party registries explicit

[stevvooe]

@flavio @vrothberg Thanks for making the need for a new flag much clearer. I agree that this is fundamentally different than the existing registry-mirrors flag.

In practice, we are mapping a namespace, which was traditionally a registry host, to a registry host. Let's come up with at least a syntax/data structure that represents this problem.

Do we have a set of example use cases that we could build a table around?

[flavio]

@vrothberg and I are working on a concrete example with config files and such. We will come back with it by the beginning of next week.

[onedr0p]

@xaleeks there's still an outstanding issue that harbor needs to fix before you can use it in that manor.

goharbor/harbor#12885

[cpuguy83]

@sudo-bmitch containerd allows passing mirror configs to the resolver. If docker does switch to use containerd's distribution code, we'd pass down these configs to containerd. This is all in the scope of the client, there is no config for mirrors for the containerd daemon itself except through the cri plugin.

[tianon]

@xaleeks I think the bigger issue with the way Harbor's implemented proxy cache is that it's too prescriptive about paths -- with the current Harbor implementation, one has to set up a proxy "project" which is then part of the namespace for the proxied images, and the user can't control or constrain that.

For example, if I set up a project of "dockerhub" on my Harbor instance and want to pull from it, I have to pull my-harbor-instance.com/dockerhub/foo/bar:baz, and I can't have the Harbor instance map, say, a project named "foo" to the "foo" namespace on Docker Hub (which would allow pulling my-harbor-instance.com/foo/bar:baz that then proxies to docker.io/foo/bar:baz).

From my understanding of what's implemented here, something like that would likely be necessary for that feature to be able to be used transparently as a pull-through mirror in this way (or this PR would need to allow for an added/adjusted image prefix on the mirror side, which would then be inconsistent with what was implemented for containerd CRI, AFAIK; https://github.com/containerd/cri/blob/bc08a19f3a44bda9fd141e6ee4b8c6b369e17e6b/docs/registry.md).

[gzm55]

@xaleeks I think the bigger issue with the way Harbor's implemented proxy cache is that it's too prescriptive about paths -- with the current Harbor implementation, one has to set up a proxy "project" which is then part of the namespace for the proxied images, and the user can't control or constrain that.

For example, if I set up a project of "dockerhub" on my Harbor instance and want to pull from it, I have to pull my-harbor-instance.com/dockerhub/foo/bar:baz, and I can't have the Harbor instance map, say, a project named "foo" to the "foo" namespace on Docker Hub (which would allow pulling my-harbor-instance.com/foo/bar:baz that then proxies to docker.io/foo/bar:baz).

From my understanding of what's implemented here, something like that would likely be necessary for that feature to be able to be used transparently as a pull-through mirror in this way (or this PR would need to allow for an added/adjusted image prefix on the mirror side, which would then be inconsistent with what was implemented for containerd CRI, AFAIK; https://github.com/containerd/cri/blob/bc08a19f3a44bda9fd141e6ee4b8c6b369e17e6b/docs/registry.md).

@tianon We met the same problem on the current Harbor, and we use a nginx reverse proxy to inject the project path, then promote a harbor project as a working registry mirror.

First, pull and push (or via proxy cache feature as @xaleeks said) the images to a dedicated harbor project and scoped by the source domain:

• alpine:latest --> harbor.company.com/registry-mirrors/docker.io/library/alpine:latest
• k8s.gcr.io/busybox:latest -> harbor.company.com/registry-mirrors/k8s.gcr.io/busybox:latest

Then create a nginx reverse proxy server at domain mirror.company.com, the main proxy rules are

  location ~ ^/v2/(?<path>.*)$ {
    if ($path != "") {
      # inject project and source domain into uri
      set $path "registry-mirrors/docker.io/$path";
    }

    proxy_pass            https://harbor.company.com/v2/$path;
    proxy_cache          off;
    proxy_set_header      Host                            $http_host;   # required for docker client's sake
    proxy_set_header      X-Real-IP                       $remote_addr; # pass on real client's IP
    proxy_set_header      X-Forwarded-For                 $proxy_add_x_forwarded_for;
    proxy_set_header      X-Forwarded-Proto               $scheme;
    proxy_read_timeout    900;
    # prepare delegate auth
    proxy_hide_header     Www-Authenticate;
    add_header            Www-Authenticate   'Bearer realm="$scheme://$http_host/service/token",service="harbor-registry"' always;
  }
  location ~ ^/service/ {
    if ($args ~* "^(.*)(scope=repository%3A)(.*)$") {
      # inject project and source domain into the auth scope
      set $args "$1$2registry-mirrors%2Fdocker.io%2F$3";
    }
    proxy_pass            https://harbor.company.com$uri$is_args$args;
    proxy_cache          off;
    proxy_set_header      Host                            $http_host;   # required for docker client's sake
    proxy_set_header      X-Real-IP                       $remote_addr; # pass on real client's IP
    proxy_set_header      X-Forwarded-For                 $proxy_add_x_forwarded_for;
    proxy_set_header      X-Forwarded-Proto               $scheme;
    proxy_read_timeout    900;
  }

Now set https://mirror.company.com as a register-mirror in docker daemon config, pull alpine:latest will be translated to harbor.company.com/registry-mirrors/docker.io/alpine:latest with correct authentication. This method can be easily extended to any third party registries.

Note the fetching blob requests on most harbor service are 307 to another storage service, so this reverse proxy should be very light.

[cpuguy83]

For people who need a solution now, https://github.com/rpardini/docker-registry-proxy/blob/master/nginx.conf is pretty handy.

You set HTTP_PROXY on dockerd and use nginx as a MITM proxy cache.
It's not a perfect solution, but it's getting me where I need to go at the moment.

[ChristianCiach]

@gzm55 This works incredibly well! Thank you!

To make this easier for other people to adopt, I've created an nginx-template that can be used with the official nginx docker image.

Let's assume you have a (public) Harbor project at https://harbor.mycompany/docker.io that is configured as a proxy-cache for docker-hub. You can then setup an nginx reverse proxy at another domain (or, as we are doing it, on the same server, but on a different port, like 5000) to use this harbor project as a transparent docker-hub proxy.

Use can use this default.conf.template for nginx:

server {
    listen ${NGINX_PORT} ssl;
    server_name ${NGINX_SERVER_NAME};
    ssl_certificate ${NGINX_SSL_CERT_FILE};
    ssl_certificate_key ${NGINX_SSL_CERT_KEY_FILE};

    # https://ssl-config.mozilla.org/#server=nginx&version=1.17.7&config=intermediate&openssl=1.1.1d&guideline=5.6
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;


    resolver 127.0.0.11;

    location ~ ^/v2/(?<path>.*)$ {
        if ($path != "") {
            # inject project and source domain into uri
            set $path "${NGINX_UPSTREAM_DOCKER_PROJECT}/$path";
        }

        proxy_pass            ${NGINX_UPSTREAM_DOCKER_REGISTRY_URL}/v2/$path;
        proxy_cache           off;
        proxy_set_header      Host                            $http_host;   # required for docker client's sake
        proxy_set_header      X-Real-IP                       $remote_addr; # pass on real client's IP
        proxy_set_header      X-Forwarded-For                 $proxy_add_x_forwarded_for;
        proxy_set_header      X-Forwarded-Proto               $scheme;
        proxy_read_timeout    900;
        # prepare delegate auth
        proxy_hide_header     Www-Authenticate;
    }
    location ~ ^/service/ {
        if ($args ~* "^(.*)(scope=repository%3A)(.*)$") {
            # inject project and source domain into the auth scope
            set $args "$1$2${NGINX_UPSTREAM_DOCKER_PROJECT}%2F$3";
        }
        proxy_pass            ${NGINX_UPSTREAM_DOCKER_REGISTRY_URL}$uri$is_args$args;
        proxy_cache           off;
        proxy_set_header      Host                            $http_host;   # required for docker client's sake
        proxy_set_header      X-Real-IP                       $remote_addr; # pass on real client's IP
        proxy_set_header      X-Forwarded-For                 $proxy_add_x_forwarded_for;
        proxy_set_header      X-Forwarded-Proto               $scheme;
        proxy_read_timeout    900;
    }

}

As you can see, there are some variables that you can set in your docker-compose.yaml, like this:

version: '3.8'

services:
  dockerhub-proxy:
    image: 'nginx:latest'
    environment:
      NGINX_PORT: '443'
      NGINX_SERVER_NAME: 'harbor.mycompany'
      NGINX_SSL_CERT_FILE: '/certs/domain-fullchain.pem'
      NGINX_SSL_CERT_KEY_FILE: '/certs/domain.key'
      NGINX_UPSTREAM_DOCKER_REGISTRY_URL: 'https://harbor.mycompany'
      NGINX_UPSTREAM_DOCKER_PROJECT: 'docker.io'
    ports:
      - '5000:443'
    volumes:
      - '/etc/YOUR_SSL_CERTS:/certs'
      - './default.conf.template:/etc/nginx/templates/default.conf.template'

In other words: Just put both files in the same directory, adjust the variables and volumes inside the docker-compose.yaml and then run docker-compose up.

To configure your docker-daemon to use this proxy, put the following into your /etc/docker/daemon.json:

{
    "registry-mirrors": ["https://harbor.mycompany:5000"]
}

I don't know why the line resolver 127.0.0.11; inside the nginx configuration is necessary, but nginx logs some errors like no resolver defined to resolve harbor.mycompany' otherwise.

[jgallucci32]

What about using DNS suffixes for the proxy cache to rewrite the URL using the subdomain as the prefix (similar to how Amazon S3 does it for buckets)? This would require a wildcard certificate and DNS entry, but that is typical for most K8s deployments these days.

For example:

• Deploy Harbor using wildcard dns entry for *.harbor.domain to point to the ingress/loadbalancer
• Create the project docker_proxy in Harbor
• Have the ingress rewrite the URL docker_proxy.harbor.domain to harbor.domain/docker_proxy

This would allow Harbor to support both private registries and mirrors at the same time.

[gzm55]

What about using DNS suffixes for the proxy cache to rewrite the URL using the subdomain as the prefix (similar to how Amazon S3 does it for buckets)? This would require a wildcard certificate and DNS entry, but that is typical for most K8s deployments these days.

For example:

• Deploy Harbor using wildcard dns entry for *.harbor.domain to point to the ingress/loadbalancer
• Create the project docker_proxy in Harbor
• Have the ingress rewrite the URL docker_proxy.harbor.domain to harbor.domain/docker_proxy

This would allow Harbor to support both private registries and mirrors at the same time.

not only the url rewrite, but also require the correct rewrite of the bearer header.

[jgallucci32]

@gzm55 Here are some code samples I wrote and tested to get nginx to rewrite the bearer token for Harbor. This assumes the project name for proxy cache is called docker.io and proxy endpoint is docker-mirror.domain.local

# Put in `http {` block
----------------------
  # Map for proxy cache
  map $upstream_http_www_authenticate $new_header {
    ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
  }
  map $args $new_args {
    ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>(?!docker.io).*)$     ${prefix2}docker.io%2F$suffix2;
  }

# Put in second `server {` block
------------------------
    # Rewrite for proxy cache
    if ($host = 'docker-mirror-domain.local') {
      rewrite ^/v2/((?!docker.io).+)$ /v2/docker.io/$1 last;
      set $args $new_args;
    }
    
# Put in `location /v2/ {` block
--------------------------------
      # Modify headers for proxy cache
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $new_header always;

[gzm55]

@gzm55 Here are some code samples I wrote and tested to get nginx to rewrite the bearer token for Harbor. This assumes the project name for proxy cache is called docker.io and proxy endpoint is docker-mirror.domain.local

# Put in `http {` block
----------------------
  # Map for proxy cache
  map $upstream_http_www_authenticate $new_header {
    ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
  }
  map $args $new_args {
    ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>(?!docker.io).*)$     ${prefix2}docker.io%2F$suffix2;
  }

# Put in second `server {` block
------------------------
    # Rewrite for proxy cache
    if ($host = 'docker-mirror-domain.local') {
      rewrite ^/v2/((?!docker.io).+)$ /v2/docker.io/$1 last;
      set $args $new_args;
    }
    
# Put in `location /v2/ {` block
--------------------------------
      # Modify headers for proxy cache
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $new_header always;

This should work, and the previous discussion gave some similar methods to play with url/header inside nginx.
Still hope this mr to be merged, then we do not need this kind of tricky work around.

[ecki]

Just setup a project in harbor as a pull through cache for your target instance and modify your docker pull command / pod specs to hit that proxy project instead.

this is useful (especially if you add some filtering/approval) but since it requires to change the container path it’s not the most transparent solution.

[vrothberg]

I think it's time to close the PR before it's hitting school age. There is a number of nice workarounds mentioned in the conversation above.

[cpuguy83]

I actually have a branch I'm working on to add support for managing "streams" which we can use to implement auth negotiation with the client.

[ChristianCiach]

@vrothberg I prefer an actual solution instead of having to use workarounds, no matter how nice they may be. As can be easily judged by the number of reactions to this issue, this issue is still valid. This PR (or a some variants) has been merged into numerous forks, and similar tools like podman properly support this use case, too. I still think docker should add support for mirrors other than those for docker hub.

Edit: Sorry. I am just noticing that you are the author of this PR, not a moby maintainer. I stand by what I said, but you are the wrong person to reply to :)