Implement private registry mirror support #34319
Conversation
|
Linking a related issue: distribution/distribution#1431 |
|
Hi @thaJeztah, thanks for your comment.
Shall I use this PR as a design proposal for #18818? We read through all related PRs before coming up with this solution, and believe that this PR matches the requirements. The most important part is that images can't be messed up as a mirror can't be used for more than one registry. |
|
CI failure is caused by one of the test vectors not being updated with the new |
|
@vrothberg Thanks for the great PR! @thaJeztah As far as I remember, we still had concerns for this model due trust about mirrors vs origin, but I am not sure where we are on this today. Let's make sure the right people are involved to make a decision here. @vrothberg Any reason we just don't accept these in the |
|
@vikstrous PTAL |
|
@stevvooe Thanks a lot for your comments.
We figured it's more explicit and expressive to configure. But if you wish, we can change that and split private registry and mirror, for instance, with an '@' in |
|
@vrothberg That is my question: is there a difference in behavior or policy, other than that these registries may need separate credentials? |
@stevvooe No, to my knowledge, there is no difference. From a mirror's perspective, it's just serving to a non-default and non-official registry URL. |
|
@vrothberg So, do you think a separate flag is really necessary? |
@stevvooe No, not from a technical point of view. I prefer a separate flag because it is more explicit, but that's just matter of taste. Do you want me to change the commit to use the |
|
@stevvooe, @thaJeztah I'd love to push this PR forward. As far as I can see, the remaining open question is whether a dedicated flag for private-registry mirrors is really needed or not. There have been some concerns if it's actually possible to find a separator that wouldn't break a valid URL [1], which is indeed really tricky. However, using There is another open PR #34495 fixing the unfortunate fact that the daemon currently ignores invalid input, but there should be no conflict as soon as the error propagation and input validation work as expected. [1] https://tools.ietf.org/html/rfc3986 |
|
@vrothberg I just LGTM'd #34495. Do you have an example of the flag format? @vikstrous Does this interfere with any other plans in this area? |
Thanks a lot, @stevvooe! In case we merge the semantics of private-registry mirrors with the So to the more I think about it, the safer I feel using a dedicated flag/config option for private-registry mirrors. |
|
There is one use case I was hoping to be able to solve that this proposal doesn't address. I think it would be valuable to be able to support mirroring hub within a sub-namespace on another registry. That way you can have a hub mirror AND local images without namespace conflicts. For, example, I would like to be able to pull |
|
I'm against using separators because the final solution looks like a hack. I really appreciate the usage of a dictionary because it's more explicit. A new flag was introduced to:
|
|
@flavio @vrothberg Thanks for making the need for a new flag much clearer. I agree that this is fundamentally different than the existing In practice, we are mapping a namespace, which was traditionally a registry host, to a registry host. Let's come up with at least a syntax/data structure that represents this problem. Do we have a set of example use cases that we could build a table around? |
|
@vrothberg and I are working on a concrete example with config files and such. We will come back with it by the beginning of next week. |
|
Let's imagine the following scenario:
This PR makes possible to satisfay this pretty common use case. The following sections will illustrate what the user has to do. Run the local docker registry mirrorsThe user has to run a vanilla docker distribution process per upstream registry that has to be mirrored. In this case the user will have to run three local docker registry: one for the Docker Hub, one for Google Container Registry and one for Quay. All the docker regitries have to be configured to act as pull through caches. For the sake of this example let's assume the user will run these local mirrors:
This is the basic configuration the user has to provide to run the Quay mirror: version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
http:
addr: :443
tls:
certificate: /certs/domain.crt
key: /certs/domain.key
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
proxy:
remoteurl: https://quay.ioThis is the basic configuration the user has to provide to run the Google Container Registry mirror: version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
http:
addr: :443
tls:
certificate: /certs/domain.crt
key: /certs/domain.key
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
proxy:
remoteurl: https://gcr.ioThis is the basic configuration the user has to provide to run the Docker Hub mirror: version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
http:
addr: :443
tls:
certificate: /certs/domain.crt
key: /certs/domain.key
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
proxy:
remoteurl: https://registry-1.docker.ioConfigure the docker hostsOn all the docker hosts the user has to run a version of the docker engine that includes this PR and create a {
"registry-mirrors": [
"https://hub-mirror.local.lan"
],
"private-registry-mirrors": {
"quay.io": "https://quay-mirror.local.lan",
"gcr.io": "https://gcr-mirror.local.lan"
}
}Profit!Once this is done all the Some examples:
Final considerationsThe code of this PR allows a pretty common scenario to be satisfied without:
The above example concentrates on a pretty common scenario made possible by this PR. Other use cases can now be achieved thanks to this PR. For example users can now mirror docker registries operated by themselves: an organization running a central docker registry inside of a cloud region could then have multiple pull through caches inside of its regional offices. |
|
What about using DNS suffixes for the proxy cache to rewrite the URL using the subdomain as the prefix (similar to how Amazon S3 does it for buckets)? This would require a wildcard certificate and DNS entry, but that is typical for most K8s deployments these days. For example:
This would allow Harbor to support both private registries and mirrors at the same time. |
not only the url rewrite, but also require the correct rewrite of the bearer header. |
|
@gzm55 Here are some code samples I wrote and tested to get nginx to rewrite the bearer token for Harbor. This assumes the project name for proxy cache is called |
This should work, and the previous discussion gave some similar methods to play with url/header inside nginx. |
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby/moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
this is useful (especially if you add some filtering/approval) but since it requires to change the container path it’s not the most transparent solution. |
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
NOTE: This is a backport/downstream patch of the upstream pull-request
for Moby, which is still subject to changes. Please visit
<moby#34319> for the current status.
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
```json
{
"registries": [
{
"Prefix": "docker.io/library/alpine",
"Mirrors": [
{
"URL": "http://local-alpine-mirror.lan"
}
]
},
{
"Prefix": "registry.suse.com",
"Mirrors": [
{
"URL": "https://remote.suse.mirror.com"
}
]
},
{
"Prefix": "http://insecure.registry.org:5000"
}
],
"registry-mirrors": ["https://deprecated-mirror.com"]
}
```
With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Signed-off-by: Valentin Rothberg <vrothberg@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
|
I think it's time to close the PR before it's hitting school age. There is a number of nice workarounds mentioned in the conversation above. |
|
I actually have a branch I'm working on to add support for managing "streams" which we can use to implement auth negotiation with the client. |
|
@vrothberg I prefer an actual solution instead of having to use workarounds, no matter how nice they may be. As can be easily judged by the number of reactions to this issue, this issue is still valid. This PR (or a some variants) has been merged into numerous forks, and similar tools like podman properly support this use case, too. I still think docker should add support for mirrors other than those for docker hub. Edit: Sorry. I am just noticing that you are the author of this PR, not a moby maintainer. I stand by what I said, but you are the wrong person to reply to :) |
|
@ChristianCiach , thank you for your excellent solution. I solved this by creating a robot account in harbor and adding a new variable NGINX_UPSTREAM_AUTHORIZATION to the compose file. This variable contains the base64 encoded name of the robot and password like this. echo 'robot$whatever:TokenGeneratedByHarbor'|base64Note the single quotes to preserver the "$" in the robots name. As I don't like removing passwords from ssl key files; I also added a variable NGINX_SSL_PASSWORD_FILE which I can use to provide the password to nginx. My template looks like this now: server {
listen ${NGINX_PORT} ssl;
server_name ${NGINX_SERVER_NAME};
ssl_certificate ${NGINX_SSL_CERT_FILE};
ssl_certificate_key ${NGINX_SSL_CERT_KEY_FILE};
ssl_password_file ${NGINX_SSL_PASSWORD_FILE};
# https://ssl-config.mozilla.org/#server=nginx&version=1.17.7&config=intermediate&openssl=1.1.1d&guideline=5.6
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
resolver 127.0.0.11;
location ~ ^/v2/(?<path>.*)$ {
if ($path != "") {
# inject project and source domain into uri
set $path "${NGINX_UPSTREAM_DOCKER_PROJECT}/$path";
}
proxy_pass ${NGINX_UPSTREAM_DOCKER_REGISTRY_URL}/v2/$path;
proxy_cache off;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Authorization "Basic ${NGINX_UPSTREAM_AUTHORIZATION}";
proxy_read_timeout 900;
# prepare delegate auth
proxy_hide_header Www-Authenticate;
}
location ~ ^/service/ {
if ($args ~* "^(.*)(scope=repository%3A)(.*)$") {
# inject project and source domain into the auth scope
set $args "$1$2${NGINX_UPSTREAM_DOCKER_PROJECT}%2F$3";
}
proxy_pass ${NGINX_UPSTREAM_DOCKER_REGISTRY_URL}$uri$is_args$args;
proxy_cache off;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Authorization "Basic ${NGINX_UPSTREAM_AUTHORIZATION}";
proxy_read_timeout 900;
}
}My compose file looks like this now: version: '3.8'
secrets:
nginx_password_file:
file: /etc/pki/tls/private/certificate.pass
services:
dockerhub-proxy:
image: 'nginx:latest'
environment:
NGINX_PORT: '443'
NGINX_SERVER_NAME: 'harbor.company'
NGINX_SSL_CERT_FILE: '/certs/certificate_joined.crt'
NGINX_SSL_CERT_KEY_FILE: '/certs/certificate.key'
NGINX_SSL_PASSWORD_FILE: '/run/secrets/nginx_password_file'
NGINX_UPSTREAM_DOCKER_REGISTRY_URL: 'https://harbor.company'
NGINX_UPSTREAM_DOCKER_PROJECT: 'dockerhub-proxy'
NGINX_UPSTREAM_AUTHORIZATION: '<base64 encoded robot secret>'
ports:
- '4443:443'
secrets:
- nginx_password_file
volumes:
- '/etc/pki/tls/certs/certificate_joined.crt:/certs/certificate_joined.crt'
- '/etc/pki/tls/private/certificate.key:/certs/certificate.key'
- '/etc/nginx/templates/default.conf.template:/etc/nginx/templates/default.conf.template'I didn't bother securing the robot credentials. Unlike the ssl certificate key, their use is quite limited. |
|
@joernott thanks caddy config or systemd |
|
We had some success with this nginx template in front of Harbor (same compose config as Becivells).
Though it seems to be about 2x slower for cache hits than accessing Harbor directly. And then Harbor in turn seems to be about 2x slower than the official Docker 'distribution' registry's pull-through cache mode. We ended up keeping that as a cache and using Harbor only as a registry. |
Add private-registry mirror support
Add support for mirroring private registries. The daemon.json config
can now be configured as exemplified below:
{ "registries": [ { "Prefix": "docker.io/alpine", "Mirrors": [ { "URL": "http://local-alpine-mirror.lan", } ] }, { "Prefix": "registry.suse.com", "Mirrors": [ { "URL": "https://remote.suse.mirror.com" } ] }, { "Prefix": "http://insecure.registry.org:5000" } ], "registry-mirrors": ["https://deprecated-mirror.com"] }With the new semantics, a mirror will be selected as an endpoint if the
specified prefix matches the prefix of the requested resource (e.g., an
image reference). In the upper example, "local-alpine-mirror" will only
serve as a mirror for docker.io if the requested resource matches the
"alpine" prefix, such as "alpine:latest" or "alpine-foo/bar".
Furthermore, private registries can now be mirrored as well. In the
example above, "remote.suse.mirror.com" will serve as a mirror for all
requests to "registry.suse.com". Notice that if no http{s,} scheme is
specified, the URI will always default to https without fallback to
http. An insecure registry can now be specified by adding the "http://"
scheme to the corresponding prefix.
Note that the configuration is sanity checked, so that a given mirror
can serve multiple prefixes if they all point to the same registry,
while a registry cannot simultaneously serve as a mirror. The daemon
will warn in case the URI schemes of a registry and one of its mirrors
do not correspond.
This change deprecates the "insecure-regestries" and "registry-mirrors"
options, while the "insecure-registries" cannot be used simultaneously
with the new "registries", which doesn't allow a fallback from https to
http for security reasons.
Signed-off-by: Flavio Castelli fcastelli@suse.com
Signed-off-by: Valentin Rothberg vrothberg@suse.com