Allow configuration of additional registries

[miminar]

Partially superseds issue #8329. It will be superseded completely with upcoming, consequent proposals based on this.

Together with extensions it obsoletes PR #10411 but is still compatible with it.

This is a root proposal for issues concerning usage and configuration of additional registries. It outlines:

1. configuration file paths
2. their format
3. their structure

Use case

As an enterprise user interacting with multiple private registries and having little or no interest in public one, I'd like to be able to pull from them without the need to specify them on command line. I'd like to specify which registries can be pushed to, which registries can be pulled from. I'd like to be able to prevent our developers from interacting with public registry or at least let Docker ask them for confirmation before a push.

In order to do this, Docker has to provide some way of configuration which would be extensible, legible and easy to edit.

Requirements

• Allow user to configure a list of additional registries specifying at least:
  1. index name
  2. index aliases
  3. associated mirrors

Proposal

Make Docker daemon read registry configuration file e.g. /etc/docker/registries.yml in YAML format.

Example

# Special object grouping auxiliary settings concerning registry configuration.
general:
    # Optional directory relative to this config file with additional
    # registries. Defaults to empty string.
    load_dir: /etc/docker/registries.d

# Special object with default properties for all registries (even those not
# configured).
defaults:

public:
    index: docker.io                # this is redundant
    aliases: [index.docker.io]      # this as well
    mirrors: [127.0.0.1:5000, docker-mirror.domain.ltd]

# Each following entry represents additional registry if it has an `index`.
redhat:
    index: registry.access.redhat.com
    mirrors: [127.0.0.1:5001]

public registry would stay hardcoded in docker for cases where registry configuration is absent or public entry is missing.

The same applies to default values. Right now there aren't any properties we could put there - they'll be specified in extension proposals.

Proposed Extensions

• Allow to specify default registries used in pull command #11816 pull from default registries
• Use a top default registry when pushing unqualified repository #11817 push to default registry
• Allow user to search multiple registries at once #11818 search default registries
• Allow user to block particular registries for certain operations #11819 block registries
• Support authentication to additional registries #11820 authentication support

[dmp42]

Hi @miminar ,

Like was said earlier, "distribution and registry" topics are discussed in the open on the distribution repo (https://github.com/docker/distribution/), on irc (#docker-distribution), and during the group meetings.

There you will find work in progress proposals to address some of your issues here, specifically about namespace and discovery:

• doc/spec: namespaces and discovery distribution/distribution#303
• Propose future approach to namespaces distribution/distribution#157

I strongly encourage you to participate in these discussions to save efforts and make sure things are happening. I'm not saying proposals and ideas here are not examined, just that being aware of (conflicting/concurrent) work in progress that is in the making is definitely useful for you.

Finally, the distribution working group does already have two persons from RedHat (@vbatts and @ncdc) that are actively participating in the discussions and design making for the registry communication pipeline. Synching/reaching out to them is probably a good idea that can help you get up to speed and figure out the best way to contribute to the community effort on these topics.

Nevertheless, about this proposal here:

As an enterprise user interacting with multiple private registries and having little or no interest in public one, I'd like to be able to pull from them without the need to specify them on command line.

If the solution to this would lead to a situation where docker pull ubuntu could mean two different things for two different persons/installs, that would very bad for the user.

It would lead to a fragmentation of the ecosystem and break the community.

This was discussed to death already, and such a solution would simply be refused.

Saving a few additional keystrokes (eg: docker pull myregistry:5000/ubuntu) is not a strong enough argument to warrant breaking the ecosystem.

Now, ability to add flexibility into image distribution (eg: uncouple image names and transport) is definitely something everybody wants, and that is being worked on: it does require "trust" so that image names can be signed to guarantee namespace consistency, though.

I'd like to specify which registries can be pushed to, which registries can be pulled from. I'd like to be able to prevent our developers from interacting with public registry or at least let Docker ask them for confirmation before a push.

From an ops perspective, if you are worried about communication to specific servers, there are simpler ways to prevent that (firewall, dns, etcetera).

Nevertheless, submitting the idea and use case to https://github.com/docker/distribution is certainly good as well.

Overall, your proposal is interesting, but:

• requires image signing so that we can guarantee the namespace is not endangered
• definitely intersect with discovery

Looking forward to talk to you more about these.

[miminar]

@dmp42 Got it. I'll join the discussions there.

[dmp42]

Thanks @miminar

Let's catchup on irc.

[LK4D4]

Discussion for distribution repo.

[fenchu]

Hi Gentlemen
This could be quite easy to fix with an environment/config variable. Would be nice to control default registry from outside the Dockerfile.

[chrrrles]

Is this proposal totally dead?

[ghost]

@dmp42

If the solution to this would lead to a situation where docker pull ubuntu could mean two different things for two different persons/installs, that would very bad for the user.

This argument just does not make sense which has been seen many times with apt, Maven, NPM, Bower and other similar packages.

"docker pull ubuntu" should mean different things when ran off user's personal laptop compared to running off corporate hardened SOEs.

"gradle dependency org.apache.lang:commons-lang" should and does mean different things when ran off user's personal laptop (just download whatever sits on maven-central) versus running off lock-down enterprise environment (usually Nexus repository manager locked down to specific / signed JAR files).

At this point we are using Nexus as a source of truth - no direct access to anything public internet from our CI build agents is the policy. This works fine with every other piece of the jigsaw puzzle, except Docker - which is extremely disappointing.

[dmp42]

"docker pull ubuntu" should mean different things when ran off user's personal laptop compared to running off corporate hardened SOEs.

Can you elaborate on why that should be so?

What would be in the ubuntu dev version that would not be in the hardened node? (or conversely)

On the contrary, as a dev, I would love to walk out of the "works locally / doesn't work in prod" mess, and code against exactly what's running in prod.

At this point we are using Nexus as a source of truth

Are you concerned about tempered content from public sources?
Or about developers using libraries you have not vetted? (and below for more questions in that line)

This argument just does not make sense which has been seen many times with apt, Maven, NPM, Bower and other similar packages.

If my code depends on library foobar in version VER, then the only way to make this reproducible is if package-manager install foobar@VER refers to the same thing everywhere you are.

If you want to fork / patch foobar, that's fine, just change your dependency and point to the fork. eg: package-manager install alternative:foobar@VER

If you want to just control distribution of the exact same thing, then it's worth asking:

• is this for "security"? and then what's wrong with content-adressability? how do you protect your local cache from poisoning? are you using TUF/DCT?
• is this for availability? then you can do that already with the registry mirroring capabilities
• is this for another reason?

Happy to chat more about this, and figuring out clear use cases (and reasons to do so).

[et304383]

How about you just implement this and let US (users) worry about the impact of configuring alternative repositories? We should be able to define a hierarchy of where to pull images from. For example:

• internal registry
• public registry

Whatever argument you (the Docker team) provide against doing this is not relevant to users who want this feature. Give us the flexibility to do what we want. Let us worry about the consequences of doing so. You can't protect us from shooting ourselves in the foot so stop trying.

This statement:

It would lead to a fragmentation of the ecosystem and break the community.

I have no idea how that would be an issue because company A wants to run an internal private registry for resolving references to internal images. Unless you can give a concrete example where providing this feature would somehow lead to a broken Docker hub, I don't see how this is a true statement.

[hdave]

The Java community has been doing this for 10+ years with its Maven repositories that hold Java JAR and WAR files -- with much success. I think it is a perfect metaphor for this situation.

[jsengla]

Trusted/signed images are an option now, so you can open this ticket back up now.

We own a licensed copy of Docker Trusted Registry running on-prem, but our hordes of developers aren't about to change all their muscle-memory and Dockerfiles to start using it overnight.

Can we pretty please have a configurable default registry for push/pull yet???
Dockerfiles are currently not portable between dev/staging/production environments specifically because of this issue.

RedHat's functional implementation of '--add-registry' and '--block-registry' means the community is extremely broken due to the availability of the exact solution that they've been begging for for years being present but only in a disparate fork of the docker engine/client. Is this the goal of Docker Incorporated developers? Doesn't this "hurt the community pretty badly"?

We just want to use docker with our own registry as a default when none are specified (kind of like the "library" repository is default, yet it may be overridden). We don't want to resort to DNS spoofing or blocking the public registry via our firewall or proxies - that would hurt the community pretty badly because we actually want our developers to have the option of pushing/pulling to/from the Docker Hub when explicitly specified.

There are YEARS worth of pull requests and forum posts and serverfault/stack exchange solutions (which break with every update to docker). There are too many rejected PRs to keep track of.
If somewhere in the "Distribution" section, this request/requirement has not been summarily dismissed, can someone please point out where the horde of wanting enterprise/government admins must now beg for this feature to be genuinely considered?

There is a solution that Red Hat implemented. It works flawlessly. You have the PR. Please consider it. I don't want to be stuck on their version of 1.10. That hurts the community.

[KedneckInc]

As a developer with 30 years experience, I see no issue with fragmentation. The average community developer, working on his or her own, is going to use docker.io by default. They have neither the need nor the desire to set up their own registry, therefore they will not experience fragmentation.

The average corporate developer, precisely where you want to maintain traction for your product, is going to set up their own registry, and it will make life vastly easier for everyone involved in the project at all levels if the default registry were configurable. A few examples:

• Developers wouldn't need to worry about remembering to use the company registry.
• People involved in IA and security would not have to worry about inadvertent use of unapproved images.
• Legal wouldn't have to worry about inadvertent pushes of proprietary images to the docker.io registry.

Without the ability to set the default registry, all of the above and more are possible. Sure, a firewall could be set up to block access to the docker.io registry, but special exceptions would have to be made in that firewall to allow pulling desired public images for placement on the local registry. Yet more complexity that could be avoided if setting the default registry were possible.

With a configurable default registry, the exception case becomes pulling from the docker.io registry, and it is clear in the Dockerfile when this is being done. This makes it much easier for everyone involved to ensure that the correct images are being used.

The argument that it will fragment the community completely ignores the needs of a significant portion of the community -- commercial users -- in favor of another portion which wouldn't even be affected by the capability, because they'd never use it.

[Levovar]

Can only agree with the last couple of statements. It is kind of ridiculous that a product tries to tell its users what features they need.
Shouldn't it be the other way around? I mean look at all the approvals of these comments asking for this -already implemented- feature.

I have never belonged to the "JAVA" community, but I still need this. It would make my life much, much easier.

So, instead of trying to tell us that "you really don't need this" maybe you should just give it to us and then don't use it on your end if you don't need it.

Ps: the situation with "registry acting as a pull-through cache of a private registry" is the exact same one. Kind of disappointing. Hope the situation/mentality of the community changes in the future

[diclophis]

Having the ability to configure the default registry seems like a huge win IMHO.

[Chrislevi]

If the solution to this would lead to a situation where docker pull ubuntu could mean two different things for two different persons/installs, that would very bad for the user.

To control the community and tell us the community whats bad/good for us is the wrong state of mind IMHO, it's our responsibility weather changing the default docker behavior or not.

We're not asking for the DOCKER TEAM to change the default behavior of Docker out of the box. we're simply asking for an option to change the default registry as ENV var or any other solutions already suggested in the distribution discussions.

If a user changes his default registry to something else, obviously
docker pull ubuntu would be different for him but it's under user's responsibility

It's a simple feature that has been asked from your own community,
which have been already implemented in a fork and obviously did NOT tear the community apart.

[KedneckInc]

@dmp42 I finally had the time to go over and look at the two threads you recommended. I'm sorry to say that I found them quite confusing, and that I don't see how/where they answer the problem that I have. To wit, for security purposes, we need to be certain that developers are, by default, (a) not pulling from a repo that has not been vetted, and (b) are not inadvertently pushing our work to a public repo.

Can you explain how any of the existing PR would cover those two issues?

The use case that you see for fragmentation does not occur in our case because our work for paying customers will never be in a public repo, so the fact that docker pull ubuntu doesn't necessarily do the same thing in this private environment that it would in the public environment is simply not a valid issue. The two environments are entirely separate.

The customers I work for have very stringent requirements for vetting before any public resource is used in the project, and would absolutely refuse to automatically use whatever new version of ubuntu came from the public repo. As such, we've had to carefully instruct our developers regarding the issues, and make sure that they never pull or push from the public repo. This would be vastly simpler if we could just set a system wide variable that set the default docker repo to our private repo. Any developer who overrides that setting would know that they are violating our security requirements.

[thaJeztah]

@KedneckInc see #34319, for which the design is currently being finalized

[SvenDowideit]

@thaJeztah @dmp42 god, this is a sad read given that Docker Inc is essentially telling us that docker pull someimage will now break based on rate limiting image manifest GET's (which, if they were not limited, would at least allow us to retain the global namespace, while then pulling the identified layer from some other source).

[finkr]

Docker supports registry mirrors (also known as "Run a Registry as a pull-through cache").
https://docs.docker.com/registry/recipes/mirror/#configure-the-docker-daemon

[steve-todorov]

"Registry mirrors" do not support authentication, which is something that ALL institutions and companies require.

#34319 , #30880

@dmp42 I strongly disagree with your entire statement.

Saving a few additional keystrokes (eg: docker pull myregistry:5000/ubuntu) is not a strong enough argument to warrant breaking the ecosystem.

This is absolutely not true, because if you using myregistry:5000/image means you will need to also re-build all images to use FROM myregistry:5000/alpine so that docker actually pulls the images from the CORRECT registry. This is specifically a gigantic problem for big organizations which have air-gapped environments in which you quite literally don't have access to registry-01.docker.io unless ALL TRAFFIC goes through a SPECIFIC registry.

[dmp42]

@steve-todorov

Five years ago :)

Either way, here is what I do to easily change my base images WITHOUT having to change ANYTHING in my dockerfiles:

Dockerfile:

ARG         BASE=docker.io
FROM        $BASE/myself/debian                                                              
RUN         echo foo

When building:

# From docker hub
docker buildx build --build-arg BASE=docker.io .
# From air-gapped registry
docker buildx build --build-arg BASE=registry.local .

Let me know if there is anything wrong with this approach.

(and yes, I'm building both airgapped and not airgapped, pushing images to multiple registries, and using different location for my base images depending on where they are built)

[testworksau]

If the solution to this would lead to a situation where docker pull ubuntu could mean two different things for two different persons/installs, that would very bad for the user.

It would lead to a fragmentation of the ecosystem and break the community.

This was discussed to death already, and such a solution would simply be refused.

Saving a few additional keystrokes (eg: docker pull myregistry:5000/ubuntu) is not a strong enough argument to warrant breaking the ecosystem.

Hi @dmp42 👋🏻

We are currently using Artifactory in our organisation. Our Docker registry has been configured with virtual repositories that consist of:

• Our own docker containers from various internal repositories
• Containers from various public registries (e.g. Docker Hub)

We'd like everyone inside our organisation to pull from our virtual repository as we also run Xray, which gives us the ability to block images that contain layers that contain vulnerabilities.

We would rather our engineers are able to set up their docker configuration once, and forget that they are pulling / pushing from / to our internal virtual repository, than have them accidentally (because they are human, not robots, and forgot to explicitly use our internal virtual repository):

• Run an unapproved / insecure container
• Waste effort using something as part of their solution that will never be approved
• Expose our internal IP to a public registry

Believe it or not, we'd also like to save those keystrokes. I don't think that's an invalid argument, particularly for those like us who have long domain names assigned to our Artifactory server, which probably takes around 10 seconds on average to type out, or find in your command history, or copy / paste from evernote.

It may seem inconsequential, but a small to mid sized corporate entity with 1000 engineers taking 10 seconds to do this once per day (cost based on $50/hr which may be too low on average) works out to be:

• 100 minutes per day wasted across the org (~$83.30)
• 8.33 hours per week (~$416)
• 433.6 hours per year (~$21,680)

Arguing that this feature shouldn't be implemented because end users could experience local problems / local fragmentation as a result of them changing the default behaviour of Docker is a bit odd to me.

The suggestion that implementing dns changes / firewall rules to achieve the same thing is "simpler" is interesting. One size does not fit all in many organisations, and there may be a valid reason to target a different docker registry with a conscious decision to implement an explicit, code reviewed FROM docker.io.... command in a dockerfile.

Can this please be reconsidered? Perhaps the option / flag / environment variable could come with documentation that warns end users of the perils of changing this setting?

[steve-todorov]

@testworksau don't waste your time on this - it's a dead end. Your engineers can simply switch to podman and configure the /etc/containers/registries.conf with:

## containers-registries.conf(5): System Registry Configuration File
[registries.search]
registries = ['your-private-repository-goes-here', 'quay.io', 'docker.io']

[registries.insecure]
registries = []

[registries.block]
registries = []

Unfortunately podman is not entirely supported in Jenkins via k8s (and i.e. Rancher), but it should work just fine for developers inside your organization otherwise. As they advertise - simply alias docker=podman. :)

[testworksau]

Thanks @steve-todorov

So it turns out @dmp42 left Docker in early 2019 so asking Olivier to change his mind is somewhat futile.

@thaJeztah @estesp @justincormack @kolyshkin @tonistiigi I appreciate that this issue has been closed for some time, however I'm wondering if you have any thoughts / different opinions.

All of the reasons given within this issue for not implementing this PR appear to have come from someone no longer working for Docker or listed as a core maintainer; I'd rather not assume you all have the same opinions / hangups as @dmp42.

#11815 (comment)

[christophermclellan]

Hello - I realise this is an incredibly belated message given this thread was created ~7 years ago ! Are there any Docker Business customers here that would be interested to try out our upcoming Registry Access Mgmt feature (currently in closed beta), which I believe fits the use cases set out in the original post.

Thanks,

Chris