Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement private registry mirror support #34319

Open
wants to merge 1 commit into
base: master
from

Conversation

@vrothberg
Copy link

@vrothberg vrothberg commented Jul 31, 2017

Add private-registry mirror support

Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:

{
"registries": [
        {
	"Prefix": "docker.io/alpine",
	"Mirrors": [
		{
			"URL": "http://local-alpine-mirror.lan",
		}
	]
        },
        {
        "Prefix": "registry.suse.com",
        "Mirrors": [
                {
                        "URL": "https://remote.suse.mirror.com"
                }
        ]
        },
        {
        "Prefix": "http://insecure.registry.org:5000"
        }
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}

With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".

Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.

Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.

This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.

Signed-off-by: Flavio Castelli fcastelli@suse.com
Signed-off-by: Valentin Rothberg vrothberg@suse.com

@vrothberg
Copy link
Author

@vrothberg vrothberg commented Jul 31, 2017

Linking a related issue: docker/distribution#1431

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jul 31, 2017

Looks like a duplicate of #32771, #21245, #19009, and #18915

Following earlier discussions / PR's on this subject, this needs a proper design first (see #19009 (comment)); I think this PR needs agreement on a design for this on the related issue: #18818

@thaJeztah thaJeztah marked this as a duplicate of #21245 Jul 31, 2017
@thaJeztah thaJeztah marked this as a duplicate of #32771 Jul 31, 2017
@vrothberg
Copy link
Author

@vrothberg vrothberg commented Aug 1, 2017

Hi @thaJeztah,

thanks for your comment.

Following earlier discussions / PR's on this subject, this needs a proper design first (see docker#19009 (comment)); I think this PR needs agreement on a design for this on the related issue: docker#18818

Shall I use this PR as a design proposal for #18818? We read through all related PRs before coming up with this solution, and believe that this PR matches the requirements. The most important part is that images can't be messed up as a mirror can't be used for more than one registry.

@cyphar
Copy link
Contributor

@cyphar cyphar commented Aug 3, 2017

CI failure is caused by one of the test vectors not being updated with the new private-registry-mirrors member.

09:36:45 ----------------------------------------------------------------------
09:36:45 FAIL: docker_cli_events_unix_test.go:390: DockerDaemonSuite.TestDaemonEvents
09:36:45 
09:36:45 [dd85c3b245ad7] waiting for daemon to start
09:36:45 [dd85c3b245ad7] daemon started
09:36:45 
09:36:45 docker_cli_events_unix_test.go:431:
09:36:45     c.Assert(out, checker.Contains, fmt.Sprintf("daemon reload %s (allow-nondistributable-artifacts=[], cluster-advertise=, cluster-store=, cluster-store-opts={}, debug=true, default-runtime=runc, default-shm-size=67108864, insecure-registries=[], labels=[\"bar=foo\"], live-restore=false, max-concurrent-downloads=1, max-concurrent-uploads=5, name=%s, registry-mirrors=[], runtimes=runc:{docker-runc []}, shutdown-timeout=10)", daemonID, daemonName))
09:36:45 ... obtained string = "2017-07-31T09:36:42.951676711Z daemon reload 2JW5:JJ5Z:E3AL:VASX:7RZB:CVIG:YWKL:URZX:TD6V:2RXB:7OMD:BXWL (allow-nondistributable-artifacts=[], cluster-advertise=, cluster-store=, cluster-store-opts={}, debug=true, default-runtime=runc, default-shm-size=67108864, insecure-registries=[], labels=[\"bar=foo\"], live-restore=false, max-concurrent-downloads=1, max-concurrent-uploads=5, name=02f072db8516, private-registry-mirrors={}, registry-mirrors=[], runtimes=runc:{docker-runc []}, shutdown-timeout=10)\n"
09:36:45 ... substring string = "daemon reload 2JW5:JJ5Z:E3AL:VASX:7RZB:CVIG:YWKL:URZX:TD6V:2RXB:7OMD:BXWL (allow-nondistributable-artifacts=[], cluster-advertise=, cluster-store=, cluster-store-opts={}, debug=true, default-runtime=runc, default-shm-size=67108864, insecure-registries=[], labels=[\"bar=foo\"], live-restore=false, max-concurrent-downloads=1, max-concurrent-uploads=5, name=02f072db8516, registry-mirrors=[], runtimes=runc:{docker-runc []}, shutdown-timeout=10)"
@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Aug 8, 2017

@vrothberg Thanks for the great PR!

@thaJeztah As far as I remember, we still had concerns for this model due trust about mirrors vs origin, but I am not sure where we are on this today. Let's make sure the right people are involved to make a decision here.

@vrothberg Any reason we just don't accept these in the --registry-mirrors flag? What is the reasoning behind the separate flag?

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Aug 8, 2017

@vikstrous PTAL

@vrothberg
Copy link
Author

@vrothberg vrothberg commented Aug 8, 2017

@stevvooe Thanks a lot for your comments.

@vrothberg Any reason we just don't accept these in the --registry-mirrors flag? What is the reasoning behind the separate flag?

We figured it's more explicit and expressive to configure. But if you wish, we can change that and split private registry and mirror, for instance, with an '@' in --registry-mirror.

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Aug 8, 2017

@vrothberg That is my question: is there a difference in behavior or policy, other than that these registries may need separate credentials?

@vrothberg
Copy link
Author

@vrothberg vrothberg commented Aug 9, 2017

@vrothberg That is my question: is there a difference in behavior or policy, other than that these registries may need separate credentials?

@stevvooe No, to my knowledge, there is no difference. From a mirror's perspective, it's just serving to a non-default and non-official registry URL.

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Aug 22, 2017

@vrothberg So, do you think a separate flag is really necessary?

@vrothberg
Copy link
Author

@vrothberg vrothberg commented Aug 23, 2017

@vrothberg So, do you think a separate flag is really necessary?

@stevvooe No, not from a technical point of view. I prefer a separate flag because it is more explicit, but that's just matter of taste.

Do you want me to change the commit to use the --registry-mirror flag instead and split host and proxy with, for instance, an @?

@vrothberg
Copy link
Author

@vrothberg vrothberg commented Sep 1, 2017

@stevvooe, @thaJeztah I'd love to push this PR forward. As far as I can see, the remaining open question is whether a dedicated flag for private-registry mirrors is really needed or not. There have been some concerns if it's actually possible to find a separator that wouldn't break a valid URL [1], which is indeed really tricky. However, using @ works as the daemon doesn't accept mirrors with http basic auth [2]. Keeping this in mind, shall I update this PR and remove the dedicated flag?

There is another open PR #34495 fixing the unfortunate fact that the daemon currently ignores invalid input, but there should be no conflict as soon as the error propagation and input validation work as expected.

[1] https://tools.ietf.org/html/rfc3986
[2] https://github.com/moby/moby/blob/master/registry/config.go#L329

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Sep 1, 2017

@vrothberg I just LGTM'd #34495.

Do you have an example of the flag format?

@vikstrous Does this interfere with any other plans in this area?

@vrothberg
Copy link
Author

@vrothberg vrothberg commented Sep 4, 2017

@vrothberg I just LGTM'd #34495.
Do you have an example of the flag format?

Thanks a lot, @stevvooe! In case we merge the semantics of private-registry mirrors with the --registry-mirror flag, we may split the mirror and the registry via mirror@registry. Currently, this should work as basic authentication (e.g., username:password@www.example.com) is not accepted (which is a good thing as passwords may be stored in clear text in the config.json). So if we go this path and decide to support basic authentication in the future, we will have a conflict with the mirror@registry syntax. Also if a user tries to use basic authentication now, it will be tricky to notify that this is not supported as we have to figure out if the specified string looks like registry@mirror or username:password@mirror.

So to the more I think about it, the safer I feel using a dedicated flag/config option for private-registry mirrors.

@vikstrous
Copy link
Contributor

@vikstrous vikstrous commented Sep 4, 2017

There is one use case I was hoping to be able to solve that this proposal doesn't address. I think it would be valuable to be able to support mirroring hub within a sub-namespace on another registry. That way you can have a hub mirror AND local images without namespace conflicts. For, example, I would like to be able to pull someone/nginx and have that effectively rewritten to registry.example.com/hubmirror/someone/nginx so that I can configure registry.example.com as a pull-through cache for hub as well as a regular registry.

@flavio
Copy link
Contributor

@flavio flavio commented Sep 5, 2017

I'm against using separators because the final solution looks like a hack. I really appreciate the usage of a dictionary because it's more explicit.

A new flag was introduced to:

  • be able to use a dictionary
  • avoid breakage of the cli for existing users (do not force the existing flag to use a dictionary)
  • make the distinction between the central hub and third party registries explicit
@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Sep 5, 2017

@flavio @vrothberg Thanks for making the need for a new flag much clearer. I agree that this is fundamentally different than the existing registry-mirrors flag.

In practice, we are mapping a namespace, which was traditionally a registry host, to a registry host. Let's come up with at least a syntax/data structure that represents this problem.

Do we have a set of example use cases that we could build a table around?

@flavio
Copy link
Contributor

@flavio flavio commented Sep 7, 2017

@vrothberg and I are working on a concrete example with config files and such. We will come back with it by the beginning of next week.

@debianmaster
Copy link

@debianmaster debianmaster commented Jul 3, 2019

+1 for this

cyphar added a commit to SUSE/docker-ce that referenced this pull request Jul 22, 2019
NOTE: This is a backport/downstream patch of the upstream pull-request
      for Moby, which is still subject to changes.  Please visit
      moby/moby#34319 for the current status.

Add support for mirroring private registries.  The daemon.json config
can now be configured as exemplified below:

```json
{
"registries": [
        {
        "Prefix": "docker.io/library/alpine",
        "Mirrors": [
                {
                        "URL": "http://local-alpine-mirror.lan"
                }
        ]
        },
        {
        "Prefix": "registry.suse.com",
        "Mirrors": [
                {
                        "URL": "https://remote.suse.mirror.com"
                }
        ]
        },
        {
        "Prefix": "http://insecure.registry.org:5000"
        }
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```

With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference).  In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".

Furthermore, private registries can now be mirrored as well.  In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com".  Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http.  An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.

Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror.  The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.

This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.

Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
cyphar added a commit to SUSE/docker-ce that referenced this pull request Jul 23, 2019
NOTE: This is a backport/downstream patch of the upstream pull-request
      for Moby, which is still subject to changes.  Please visit
      moby/moby#34319 for the current status.

Add support for mirroring private registries.  The daemon.json config
can now be configured as exemplified below:

```json
{
"registries": [
        {
        "Prefix": "docker.io/library/alpine",
        "Mirrors": [
                {
                        "URL": "http://local-alpine-mirror.lan"
                }
        ]
        },
        {
        "Prefix": "registry.suse.com",
        "Mirrors": [
                {
                        "URL": "https://remote.suse.mirror.com"
                }
        ]
        },
        {
        "Prefix": "http://insecure.registry.org:5000"
        }
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```

With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference).  In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".

Furthermore, private registries can now be mirrored as well.  In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com".  Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http.  An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.

Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror.  The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.

This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.

Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
@xaleeks
Copy link

@xaleeks xaleeks commented Aug 14, 2019

Is there an official update on the progress for this from Moby team? It's still a highly desirable feature

@ccll
Copy link

@ccll ccll commented Aug 15, 2019

@zhouhaibing089 Nice and sweet hint on the 'containerd' runtime, I've tried that on vanilla k8s and works perfectly.
Unfortunately our production k8s environment is Openshift Origin (OKD) which does not have support for 'containerd', so the native implementation in 'dockerd' is our last hope.

@ccll
Copy link

@ccll ccll commented Sep 9, 2019

@zhouhaibing089 Would you mind sharing your setup of using containerd with CRI plugin?
I've been using containerd with CRI plugin successfully on vanilla Kubernetes, recently I need to work on a docker swarm cluster, and failed miserably.

I know nothing about the internal of docker/containerd/cri, but I read somewhere modern docker already depends on containerd, so I did some experiments on docker-containerd.
I can tell that "/etc/containerd/config.toml" is actually load by containerd as malformed config will fail the process, but CRI plugin config seems have no effect on image pushing/pulling, I guess the mechanism behind docker-containerd is not the same as k8s-containerd?

Would any one please give some advice on this situation, it's possible or impossible (to use containerd CRI plugin registry mirrors together with docker or docker swarm)?

@stevenvanryckeghem
Copy link

@stevenvanryckeghem stevenvanryckeghem commented Apr 2, 2020

Any news on this matter? Would be a very nice feature to have...

@r-sniper
Copy link

@r-sniper r-sniper commented May 6, 2020

Any update? There are use cases where this feature is necessary

@DanielFallon
Copy link

@DanielFallon DanielFallon commented Jun 22, 2020

@r-sniper
FYI, if you don't need to be using moby/Docker, others have mentioned that this has been implemented in other container runtimes even if moby has been bikeshedding on this since 2017. I've been following this pull request/related issue since I worked at a financial firm years ago, and it doesn't seem like fixing the original sin of conflating image namespace with registry hostname is much of a priority. (@vrothberg the current design looks fabulous btw)

note: reiterating this with better formatting to help with Readability/SEO, if anyone can think of others I'll happily add them to my list :)

Podman/Buildah/Libpod/CRI-O

added last year in containers/image#564 by @saschagrunert
^it was coincidentally reviewed by Valentin 😂
See related documentation:
https://github.com/containers/image/blob/master/docs/containers-registries.conf.5.md

containerd

added 2 years ago in containerd/cri#531 by @abhi
See related documentation:
https://github.com/containerd/cri/blob/master/docs/registry.md#configure-registry-endpoint

@patsevanton
Copy link

@patsevanton patsevanton commented Jun 23, 2020

@DanielFallon how to use Podman/Buildah/Libpod/CRI-O to create Kubernetes (Install by Kubespray)?

@cpuguy83
Copy link
Contributor

@cpuguy83 cpuguy83 commented Oct 2, 2020

How about we just merge this as "experimental", @thaJeztah

There is a real need here.

@sudo-bmitch
Copy link

@sudo-bmitch sudo-bmitch commented Oct 2, 2020

I'd like to see this added too. Curious what Docker's plans are for this since containerd has an implementation there. Is Docker thinking of moving to containerd for push/pull requests, or is there a desire to refresh this PR for the current code base. I'd be happy to put in some effort for the latter.

@xaleeks
Copy link

@xaleeks xaleeks commented Oct 2, 2020

I wanted to share with everyone on this thread that Harbor latest release v2.1 supports proxy cache for docker registry now, and allow for more 3rd party registries to be supported soon. Just setup a project in harbor as a pull through cache for your target instance and modify your docker pull command / pod specs to hit that proxy project instead. Genuinely curious if we made progress in the right direction and appreciate any feedback.

@onedr0p
Copy link

@onedr0p onedr0p commented Oct 2, 2020

@xaleeks there's still an outstanding issue that harbor needs to fix before you can use it in that manor.

goharbor/harbor#12885

@cpuguy83
Copy link
Contributor

@cpuguy83 cpuguy83 commented Oct 4, 2020

@sudo-bmitch containerd allows passing mirror configs to the resolver. If docker does switch to use containerd's distribution code, we'd pass down these configs to containerd. This is all in the scope of the client, there is no config for mirrors for the containerd daemon itself except through the cri plugin.

@tianon
Copy link
Member

@tianon tianon commented Oct 6, 2020

@xaleeks I think the bigger issue with the way Harbor's implemented proxy cache is that it's too prescriptive about paths -- with the current Harbor implementation, one has to set up a proxy "project" which is then part of the namespace for the proxied images, and the user can't control or constrain that.

For example, if I set up a project of "dockerhub" on my Harbor instance and want to pull from it, I have to pull my-harbor-instance.com/dockerhub/foo/bar:baz, and I can't have the Harbor instance map, say, a project named "foo" to the "foo" namespace on Docker Hub (which would allow pulling my-harbor-instance.com/foo/bar:baz that then proxies to docker.io/foo/bar:baz).

From my understanding of what's implemented here, something like that would likely be necessary for that feature to be able to be used transparently as a pull-through mirror in this way (or this PR would need to allow for an added/adjusted image prefix on the mirror side, which would then be inconsistent with what was implemented for containerd CRI, AFAIK; https://github.com/containerd/cri/blob/bc08a19f3a44bda9fd141e6ee4b8c6b369e17e6b/docs/registry.md).

@gzm55
Copy link

@gzm55 gzm55 commented Oct 11, 2020

@xaleeks I think the bigger issue with the way Harbor's implemented proxy cache is that it's too prescriptive about paths -- with the current Harbor implementation, one has to set up a proxy "project" which is then part of the namespace for the proxied images, and the user can't control or constrain that.

For example, if I set up a project of "dockerhub" on my Harbor instance and want to pull from it, I have to pull my-harbor-instance.com/dockerhub/foo/bar:baz, and I can't have the Harbor instance map, say, a project named "foo" to the "foo" namespace on Docker Hub (which would allow pulling my-harbor-instance.com/foo/bar:baz that then proxies to docker.io/foo/bar:baz).

From my understanding of what's implemented here, something like that would likely be necessary for that feature to be able to be used transparently as a pull-through mirror in this way (or this PR would need to allow for an added/adjusted image prefix on the mirror side, which would then be inconsistent with what was implemented for containerd CRI, AFAIK; https://github.com/containerd/cri/blob/bc08a19f3a44bda9fd141e6ee4b8c6b369e17e6b/docs/registry.md).

@tianon We met the same problem on the current Harbor, and we use a nginx reverse proxy to inject the project path, then promote a harbor project as a working registry mirror.

First, pull and push (or via proxy cache feature as @xaleeks said) the images to a dedicated harbor project and scoped by the source domain:

  • alpine:latest --> harbor.company.com/registry-mirrors/docker.io/library/alpine:latest
  • k8s.gcr.io/busybox:latest -> harbor.company.com/registry-mirrors/k8s.gcr.io/busybox:latest

Then create a nginx reverse proxy server at domain mirror.company.com, the main proxy rules are

  location ~ ^/v2/(?<path>.*)$ {
    if ($path != "") {
      # inject project and source domain into uri
      set $path "registry-mirrors/docker.io/$path";
    }

    proxy_pass            https://harbor.company.com/v2/$path;
    proxy_cache          off;
    proxy_set_header      Host                            $http_host;   # required for docker client's sake
    proxy_set_header      X-Real-IP                       $remote_addr; # pass on real client's IP
    proxy_set_header      X-Forwarded-For                 $proxy_add_x_forwarded_for;
    proxy_set_header      X-Forwarded-Proto               $scheme;
    proxy_read_timeout    900;
    # prepare delegate auth
    proxy_hide_header     Www-Authenticate;
    add_header            Www-Authenticate   'Bearer realm="$scheme://$http_host/service/token",service="harbor-registry"' always;
  }
  location ~ ^/service/ {
    if ($args ~* "^(.*)(scope=repository%3A)(.*)$") {
      # inject project and source domain into the auth scope
      set $args "$1$2registry-mirrors%2Fdocker.io%2F$3";
    }
    proxy_pass            https://harbor.company.com$uri$is_args$args;
    proxy_cache          off;
    proxy_set_header      Host                            $http_host;   # required for docker client's sake
    proxy_set_header      X-Real-IP                       $remote_addr; # pass on real client's IP
    proxy_set_header      X-Forwarded-For                 $proxy_add_x_forwarded_for;
    proxy_set_header      X-Forwarded-Proto               $scheme;
    proxy_read_timeout    900;
  }

Now set https://mirror.company.com as a register-mirror in docker daemon config, pull alpine:latest will be translated to harbor.company.com/registry-mirrors/docker.io/alpine:latest with correct authentication. This method can be easily extended to any third party registries.

Note the fetching blob requests on most harbor service are 307 to another storage service, so this reverse proxy should be very light.

@cpuguy83
Copy link
Contributor

@cpuguy83 cpuguy83 commented Oct 12, 2020

For people who need a solution now, https://github.com/rpardini/docker-registry-proxy/blob/master/nginx.conf is pretty handy.

You set HTTP_PROXY on dockerd and use nginx as a MITM proxy cache.
It's not a perfect solution, but it's getting me where I need to go at the moment.

innovate-invent pushed a commit to brinkmanlab/cloud_recipes that referenced this pull request Oct 16, 2020
@ChristianCiach
Copy link

@ChristianCiach ChristianCiach commented Nov 2, 2020

@gzm55 This works incredibly well! Thank you!

To make this easier for other people to adopt, I've created an nginx-template that can be used with the official nginx docker image.

Let's assume you've a (public) Harbor project at https://harbor.mycompany/docker.io that is configured as a proxy-cache for docker-hub. You can then setup an nginx reverse proxy at another domain (or, as we are doing it, on the same server, but on a different port, like 5000) to use this harbor project as a transparent docker-hub proxy.

Use can use this default.conf.template for nginx:

server {
    listen ${NGINX_PORT} ssl;
    server_name ${NGINX_SERVER_NAME};
    ssl_certificate ${NGINX_SSL_CERT_FILE};
    ssl_certificate_key ${NGINX_SSL_CERT_KEY_FILE};

    # https://ssl-config.mozilla.org/#server=nginx&version=1.17.7&config=intermediate&openssl=1.1.1d&guideline=5.6
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;


    resolver 127.0.0.11;

    location ~ ^/v2/(?<path>.*)$ {
        if ($path != "") {
            # inject project and source domain into uri
            set $path "${NGINX_UPSTREAM_DOCKER_PROJECT}/$path";
        }

        proxy_pass            ${NGINX_UPSTREAM_DOCKER_REGISTRY_URL}/v2/$path;
        proxy_cache           off;
        proxy_set_header      Host                            $http_host;   # required for docker client's sake
        proxy_set_header      X-Real-IP                       $remote_addr; # pass on real client's IP
        proxy_set_header      X-Forwarded-For                 $proxy_add_x_forwarded_for;
        proxy_set_header      X-Forwarded-Proto               $scheme;
        proxy_read_timeout    900;
        # prepare delegate auth
        proxy_hide_header     Www-Authenticate;
    }
    location ~ ^/service/ {
        if ($args ~* "^(.*)(scope=repository%3A)(.*)$") {
            # inject project and source domain into the auth scope
            set $args "$1$2${NGINX_UPSTREAM_DOCKER_PROJECT}%2F$3";
        }
        proxy_pass            ${NGINX_UPSTREAM_DOCKER_REGISTRY_URL}$uri$is_args$args;
        proxy_cache           off;
        proxy_set_header      Host                            $http_host;   # required for docker client's sake
        proxy_set_header      X-Real-IP                       $remote_addr; # pass on real client's IP
        proxy_set_header      X-Forwarded-For                 $proxy_add_x_forwarded_for;
        proxy_set_header      X-Forwarded-Proto               $scheme;
        proxy_read_timeout    900;
    }

}

As you can see, there are some variables that you can set in your docker-compose.yaml, like this:

version: '3.8'

services:
  dockerhub-proxy:
    image: 'nginx:latest'
    environment:
      NGINX_PORT: '443'
      NGINX_SERVER_NAME: 'harbor.mycompany'
      NGINX_SSL_CERT_FILE: '/certs/domain-fullchain.pem'
      NGINX_SSL_CERT_KEY_FILE: '/certs/domain.key'
      NGINX_UPSTREAM_DOCKER_REGISTRY_URL: 'https://harbor.mycompany'
      NGINX_UPSTREAM_DOCKER_PROJECT: 'docker.io'
    ports:
      - '5000:443'
    volumes:
      - '/etc/YOUR_SSL_CERTS:/certs'
      - './default.conf.template:/etc/nginx/templates/default.conf.template'

In other words: Just put both files in the same directory, adjust the variables and volumes inside the docker-compose.yaml and then run docker-compose up.

To configure your docker-daemon to use this proxy, put the following into your /etc/docker/daemon.json:

{
    "registry-mirrors": ["https://harbor.mycompany:5000"]
}

I don't know why the line resolver 127.0.0.11; inside the nginx configuration is necessary, but nginx logs some errors like no resolver defined to resolve harbor.mycompany' otherwise.

@jgallucci32
Copy link

@jgallucci32 jgallucci32 commented Nov 11, 2020

What about using DNS suffixes for the proxy cache to rewrite the URL using the subdomain as the prefix (similar to how Amazon S3 does it for buckets)? This would require a wildcard certificate and DNS entry, but that is typical for most K8s deployments these days.

For example:

  • Deploy Harbor using wildcard dns entry for *.harbor.domain to point to the ingress/loadbalancer
  • Create the project docker_proxy in Harbor
  • Have the ingress rewrite the URL docker_proxy.harbor.domain to harbor.domain/docker_proxy

This would allow Harbor to support both private registries and mirrors at the same time.

@gzm55
Copy link

@gzm55 gzm55 commented Nov 11, 2020

What about using DNS suffixes for the proxy cache to rewrite the URL using the subdomain as the prefix (similar to how Amazon S3 does it for buckets)? This would require a wildcard certificate and DNS entry, but that is typical for most K8s deployments these days.

For example:

  • Deploy Harbor using wildcard dns entry for *.harbor.domain to point to the ingress/loadbalancer
  • Create the project docker_proxy in Harbor
  • Have the ingress rewrite the URL docker_proxy.harbor.domain to harbor.domain/docker_proxy

This would allow Harbor to support both private registries and mirrors at the same time.

not only the url rewrite, but also require the correct rewrite of the bearer header.

@jgallucci32
Copy link

@jgallucci32 jgallucci32 commented Nov 12, 2020

@gzm55 Here are some code samples I wrote and tested to get nginx to rewrite the bearer token for Harbor. This assumes the project name for proxy cache is called docker.io and proxy endpoint is docker-mirror.domain.local

# Put in `http {` block
----------------------
  # Map for proxy cache
  map $upstream_http_www_authenticate $new_header {
    ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
  }
  map $args $new_args {
    ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>(?!docker.io).*)$     ${prefix2}docker.io%2F$suffix2;
  }

# Put in second `server {` block
------------------------
    # Rewrite for proxy cache
    if ($host = 'docker-mirror-domain.local') {
      rewrite ^/v2/((?!docker.io).+)$ /v2/docker.io/$1 last;
      set $args $new_args;
    }
    
# Put in `location /v2/ {` block
--------------------------------
      # Modify headers for proxy cache
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $new_header always;
@gzm55
Copy link

@gzm55 gzm55 commented Nov 13, 2020

@gzm55 Here are some code samples I wrote and tested to get nginx to rewrite the bearer token for Harbor. This assumes the project name for proxy cache is called docker.io and proxy endpoint is docker-mirror.domain.local

# Put in `http {` block
----------------------
  # Map for proxy cache
  map $upstream_http_www_authenticate $new_header {
    ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
  }
  map $args $new_args {
    ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>(?!docker.io).*)$     ${prefix2}docker.io%2F$suffix2;
  }

# Put in second `server {` block
------------------------
    # Rewrite for proxy cache
    if ($host = 'docker-mirror-domain.local') {
      rewrite ^/v2/((?!docker.io).+)$ /v2/docker.io/$1 last;
      set $args $new_args;
    }
    
# Put in `location /v2/ {` block
--------------------------------
      # Modify headers for proxy cache
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $new_header always;

This should work, and the previous discussion gave some similar methods to play with url/header inside nginx.
Still hope this mr to be merged, then we do not need this kind of tricky work around.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.