Registry V2 mirror proposal

[RichardScothern]

Milestones

• Client changes
• Cache-control headers
• Image and Manifest pull-through
  • Serialized remote blob fetch
  • Redirect handling
• Config
• Auth
• Cache expiration
• Namespace integration / access control implementation
• Unit tests
• Integration tests
• LGTM from @dmcgowan @stevvooe @aaronlehmann

Proposed for #19

Registry V2 mirror proposal

A registry should be able to operate as a pull-through mirroring cache. This means that if a pull operation cannot proceed locally due to missing content, the registry should be able to defer the request to an upstream registry.

Unlike the V1 registry mirroring feature, a V2 registry mirror can proxy to a private registry. This will allow more flexible registry topologies.

Behavior

from Issue #19:

Pull

When pulling a manifest, if the content is available locally, it will be served as is. Optionally, the local may check the remote if the content has been updated, using conditional http requests and update the local content. If it is not available locally, the request should be forwarded to the remote registry. If the remote request proceeds, the manifest should be stored locally then served in response to the local request.

When pulling a blob, if the content is available locally, serve it as is. If not available locally, forward the request to the remote registry. If the data is directly available, the data should be forwarded to the client and stored locally, concurrently. If the remote issues a redirect, the local registry should download the data into the local cache and serve the data directly.

Push

All push operations are only attempted on the "local" registry. If they fail, they will not be forwarded to the remote registry.

Docker Daemon control

As with the v1 registry, the Docker daemon will supply the --registry-mirror option to make requests through the proxying registry.

Implementation

HTTP Cache Headers

Cache-Control

The value in this header specifies how long the data associated with the request will be cached for. The TTL will be obeyed by the registry mirror and retained with the response to the client in order to take advantage of intermediate web caches. The TTL will be specified by the max-age directive.

If-None-Match

The value of this header will be used by the registry mirror to check for updated images in the back-end registry.

Etag

An Etag header will be used in combination with the If-None-Match header. The value of the Etag header will be the same as Docker-Content-Digest

TTL & Flushing

A registry mirror will obey TTLs specified for mutable content. When the TTL for a given resource has expired it will be removed from the mirror and fetched from the upstream registry on the subsequent request.

Immutable data does not have to obey TTL as it can never become invalidated and thus will remain in the cache.

Credentials

A registry mirror will not forward the credentials from the client request. A registry mirror will have its own credentials specified in the configuration. A registry mirror will send an altered User-Agent HTTP header to identify itself as a mirror. Identification of the original client can be forwarded by the mirror using the X-Forwarded-For HTTP header.

Registry client Behavior

Much of the client behavior will be defined in the registry client package (see dependencies). However the following behavior will be important to ensure good cache performance.

• Conditional checks

  The local registry will avoid unnecessary data transfers by issuing conditional HTTP requests to the remote registry. This is especially important for large layers. In order to avoid unnecessary data transfer, an Etag HTTP header (see above) will be used to check freshness.

• Concurrent layer pulling

  Where a set of blobs is known at request time, the individual layers will be downloaded to the mirror concurrently. In the future, batching these requests will provide a greater performance gain.

Access Control

A registry mirror will be configurable with an access control list which will give fine grained control over which source images are served from.

Access control will consists image names with wildcard cards (including tags), and the value will specify wether to only serve from the local or the upstream registry.

e.g.

acl:
   'library/*': "remote"
   'mycorp/product': "remote"
   'mycorp/product:testing': "local"

Compression

In order to reduce bandwidth on a network, the Registry will support transport encoding. The Registry mirror will set the HTTP Accept Encoding header with supported compression algorithms and the Registry backend will optionally use this to compress certain layers

Configuration

The following aspects of the Caching behavior will be supported in the config file.

Option	Meaning
upstream	URL of upstream registry
acl	see example above
Username	Username to present to remote registry
Password	Password to present to remote registry

Integration with the Layer Cache

The mirror cache belonging to the registry mirror will be populated with data pulled from the remote registry.

Circular Dependencies

Allowing proxying to arbitrary registry open up the possibility of circular dependencies. Request sources can be tracked in a number of different ways:

• The number of proxy hops can be tracked and capped at an arbitrary number
• The proxies involved in a request can be tracked in an HTTP header (e.g. X-Forwarded-For) and the request terminated when a circle is detected.

Code Dependencies

Implementation is dependent on some current pull requests. A hard requirement is a stable registry client.

• #387 Add client implementation of distribution interface

Other pull request which are preferable to reduce code churn include
various refactor pull requests listed #447

[munhitsu]

1. Can you extend the proposal to mention flow when 2 clients request the same layer simultaneously.
   I would expect it to be a quite typical flow. Something similar to:
   • Client A requests layer X from mirror Alpha
   • mirror Alpha starts download and feeds data to client A as data arrives while also storing the data on a drive
   • Client B requests layer X from mirror Alpha while it's not yet fully cached
   • mirror Alpha delivers to client A all cached so far data and starts feeding to client A data as it is being downloaded from upstream
   • A & B finished download
2. Registry Audit Trail
   It would be preferred to keep in the main registry a record on who was the original creator of layer. I know it's mirror authenticating but in the end we want to know who created the image. It's not clear in the original proposal.

CC: @stevvooe

[dmcgowan]

A few things to note...

Related to the If-Modified-Since header, this statement "If these headers are note set by the client, the Registry mirror will set them with a predefined set of values." doesn't apply if the client being discussed is the daemon. The daemon could be able to set the cache control header, but the If-Modified-Since would only apply to interactions between the proxy-cache and upstream. The registry should be updated to check its own last-modified and return a 304 Not Modified when the HEAD request comes in from a proxy-cache.

For configuration, the upstream and ACL configuration should be managed completely by the namespace logic (which includes upstream discovery). The namespace logic will be integrated into the client and the proxy-cache should rely on this. From the daemon perspective, specifying a mirror will override the default remote discovery process to always interact with the mirror.

We need to make a decision on how the proxy-cache will interact with V1 registries. If the daemon is always going to the mirror, then either the proxy-cache should cache v1 content (this would be awesome), or it will need to ignore v1 configurations and ensure the client still falls back to v1. This does raise the question of how the registry mirror flag will interact when it is intending to use a v1 mirror. The last issue we may be able to break the behavior (we will have to eventually anyways since it never should have been merged and is horribly insecure), but it will be a battle.

[RichardScothern]

@munhitsu : Thank you for the feedback. Behavior to ensure simultaneous requests from duplicating data movement will certainly be considered. To stay short and easily digestible, this document is for non-functional requirements only. Regarding point 2: We see image and layer provenance ultimately being provided by our trust service, which is being actively worked on.

[RichardScothern]

@dmcgowan
👍 for the HTTP header comment, I've updated the proposal.

I largely agree on the ACL/namespace, but we should have a discussion.

Registry V1 interaction had not crossed my mind, and will require some thought.

[dmcgowan]

After discussion about v1, the conclusion we came to what that the v2 mirror should only support v2 upstreams and the client will be responsible for contacting v1 if no data was found by the v2 mirror.

[ahmetb]

@RichardScothern thanks, this looks nice overall.

My question is about Cache-Control header: As far as I understand Cache-Control will be used to set TTL for the blobs that are cached on the mirror by the clients. If that's the case, in the public mirrors where everybody can pull layers from, won't the clients be able to set TTL for layers on the mirror and won't multiple users be able to set different TTLs for the same blob that can conflict with each other?

[RichardScothern]

Hi @ahmetalpbalkan . TTL will be set by Cache-Control, but by the upstream registry, not the clients. I had initially considered allowing clients to set this (and wrote this in the initial version of the proposal), but changed it for the reasons you describe.

The TTL will be initially a pre-canned value (~2 weeks), but possibly configurable in later versions.

[ahmetb]

@RichardScothern ah awesome, thanks for the answer.

[stevvooe]

@RichardScothern Took a solid look at this and took notes along the way. They are available in full below. Here is a summary of what needs to be done:

• Make a PR to get the ETag, Cache-Control and conditionals correct. This may only be unit tests for confirmation, but let's make sure we have the right behavior in the upstream.
• Explore the possibility of making a registry readonly using acls. I put a strawman configuration below. This may be too much but it could be pretty interesting.

Here are my notes:

• While the content is downloading locally, will the registry redirect the
  primary source or hold the request until the content is available?
• How will TTLs be managed?
• Separate PR for etag, cache-control, etc. Let's get this started right away.
• Let's also add a registry readonly mode. Can this be part of ACL?
• How will this work with drivers that implement URLFor? Should this be
  disabled or should we allow disabling it?

A registry mirror will send an altered User-Agent HTTP header to identify
itself as a mirror. Identification of the original client can be forwarded
by the mirror using the X-Forwarded-For HTTP header.

I am not sure why this is necessary.

• Will the ACL functionality also control pull/push? Something like this:

registries: # maybe call this registries?
  remote:
    # Follow convention from notification endpoints here.
    type: remote
    cache: local # Allows one to configure whether or not the data is stored in the local cache
    url: https://registry-1.docker.com/v2
    auth:
      type: basic
      username: foo
      password: bar
    actions: ["pull"]
  production:
    type: remote
    url: https://production.internal/v2
    auth:
      type: basic
      username: foo
      password: bar
    actions: ["pull"]
  local:
    type: storage
    actions: ["pull"] # disables pushes to local
routes:
  library/*: "remote"
  mycorp/product: "production"
  mycorp/product:latest: "local"

[ahmetb]

In addition to @stevvooe's point:

• While the content is downloading locally, will the registry redirect the primary source or hold the request until the content is available?

... Or download and stream back on the request at the same time?

[dmcgowan]

@stevvooe

Will the ACL functionality also control pull/push? Something like this:

Some of this configuration could overlap with the namespace configuration. Would you expect this configuration to be used to configure the namespace, in which case it would be useful to provide a scope for a given registry.

[stevvooe]

@dmcgowan There is definitely some crossover here. And while I despise the evil yaml, the above config does a fantastic job of conveying what is available and how to connect. That section could adopt the term "endpoint" and work without issue.

[RichardScothern]

@stevvooe

While the content is downloading locally, will the registry redirect the primary source or hold the request until the content is available?

I think the registry should block the request while the content is downloading. The download to mirror from remote should never be slower than downloading to local from remote, and this is the simplest implementation. @ahmetalpbalkan , I would consider simultaneous stream splitting for the second milestone.

How will TTLs be managed?

I imagine something along the lines of: when a repo is mirrored to the cache its TTL is written in a file. A background process can periodically check TTL files and remove the repositories. This process and client reads will need to be synchronized with a RW lock.

How will this work with drivers that implement URLFor? Should this be disabled or should we allow disabling it?

Disable it.

A registry mirror will send an altered User-Agent HTTP header to identify itself as a mirror. Identification of the original client can be forwarded by the mirror using the X-Forwarded-For HTTP header.

If we want to track the real IPs which eventually hit the hub, we will need to adhere to X-Forwarded-For, otherwise we will just get mirror IP addresses.

@stevvooe @dmcgowan : whatever ACL we have should be namespace compatible. I think the 'routes' and 'actions' in your strawman yaml should be namespace artifacts.

[ahmetb]

@RichardScothern if the mirror blocks while downloading the "cache miss" layer without streaming at the same time as it downloads, many HTTP clients would just timeout by default due to in activity. So I would expect non-docker cli clients to have this bug by default (b/c they use standard http clients, all of which has a default timeout configured).

[RichardScothern]

That is a great point @ahmetalpbalkan , the socket can't be idle for that long. That makes the decision easier :)

[ahmetb]

@RichardScothern that aside, for instance in Microsoft Azure we'd like to host Docker Hub Mirrors, one per datacenter, simply because we want images to be pulled down faster with in-DC networking. By holding off until the image gets fetched from centralized source to the mirror, you serialize the process and make it "t_total= (t_{download from mirror} + t_{stream to client})". For that I believe we have 3 options in the case of "cache miss":

• Block the client until the download finishes (assuming clients won't timeout in the meanwhile)
• Redirect the client to the source URL (Docker Hub blob) and kick off a download in the background (to populate the cache for the next time)
• Download and stream at the same time (sounds like the best solution and easily doable to me by using a io.MultiWriter?)

[lox]

A project of mine has a Cache-Control parser and tests https://github.com/lox/httpcache/blob/master/cachecontrol.go

I'll see if I can repurpose some of the stuff in there for this PR.

[mattmoor]

I know I'm late to the party, but I'd +1 @ahmetalpbalkan's point.

IMO, the mirror shouldn't pull-through on misses, it should 404 and the client should get the feed directly from the source, and the mirror should queue its own async fetch of the data.

This creates a nice separation of concerns and gives a nice answer for how you deal with concurrent pulls of an uncached layer (all get a 404 until it's cache-resident).

However, this is also only a problem for on-demand mirroring. Has anyone considered providing a design for pro-active mirroring (listening for changes on the Hub)?

Another more fundamental issue I had with the v1 mirroring scheme was that it required "all or nothing" hits, effectively mandating pull-through as the implementation, or you couldn't benefit from partial hits.

What I mean is that the registry's layer storage forms a tree with a lot of base layer sharing (e.g. debian, ubuntu, redhat, ...), but to get a full hit, you have to cache (or pull-through) the incredibly high-degree fringe of this tree.

[bobrik]

@mattmoor downloading an image from 1000 machines with 404 on miss essentially kills any benefit from mirroring in many cases. If you only need image once on deploy, you'll have 0% hit rate.

Let's do some math with 1000 hosts, 1 image of 1gb.

1. Pull-through: 1gb to download from upstream (docker hub), X seconds.
2. With 404 on miss: 1000gb to download from upstream (docker hub), 1000X seconds.

This is the worst-case scenario when one client can saturate internet link. Believe it or not, but I've seen production systems with hundreds of hosts where pulls from docker hub where at 14kb/s.

Download and stream at the same time (sounds like the best solution and easily doable to me by using a io.MultiWriter?)

This one is promising.

[mattmoor]

@bobrik I'm all too familiar with this scenario.

My point is that the specification for mirroring should not mandate pull-though. A given implementation of the spec may pull through, but the spec and client-behavior should allow for alternate implementations.

In topologies such as what you describe, you would deploy a fairly aggressive strategy.

For public cloud providers, who are primarily interested in mirroring common base layers in DCs, we can operate as I describe.

@stevvooe I'd love to see proposals like this enumerate Goals / non-Goals so we can better frame these discussions?

[ahmetb]

@bobrik thanks for supporting my suggestion. Here's a bit more context on what we will do (as a cloud provider, Azure):

Assuming the "download and stream at the same time in case of cache miss" is implemented, we will have bots to pull most popular N images from Docker Hub every hour or so, to fill up the cache eagerly with the missing layers and reduce cache misses statistically as Matt said.

@mattmoor +1 on enumerating goals and steps, timelines to prevent this proposal from stalling.

[stevvooe]

@mattmoor @ahmetalpbalkan This proposal may be outside of either of your particular use cases. Let's review the problem statement:

A registry should be able to operate as a pull-through mirroring cache. This means that if a pull operation cannot proceed locally due to missing content, the registry should be able to defer the request to an upstream registry.

This particular feature has been pretty narrowly scoped from the start. #19 and the comments provide a clear discussion of the scope and was submitted at the start of January.

"Pro-active mirroring", as you call it, is outside the scope of this proposal. This proposal also does not prevent anyone from building such a mirroring system. Albeit, we are deferring the implementation of "pro-active mirroring" or mirroring, as we are calling, to when the trust subsystem is in place, allowing one to verify trusted content from untrusted sources (such as mirrors).

The nice thing about this proposal is there is no entry point for the specification and only a requirement in the daemon to redirect requests through a registry of a different name (ie send "library/ubuntu" to "cache.internal/library/ubuntu"). This proposal only affects features of the registry and the actual implementation is left to the implementor. All of the suggested strategies could be attempted on top of these primitives.

The important aspect to consider is this caching strategy covers the narrow use case of someone running a local registry cache to ensure hits on remote pulls. The edge cases provided are problematic but not having this feature is worse. For small deployments, this will perform well and save bandwidth. For large deployments of new upstreams, a "warming pull" could be coordinated if dog pile is a problem. More sophisticated handling of dog-pile and pull-through could be part of another PR.

Considering the time this proposal has been available and the lack of actionable counter-proposals that cover this use case, I don't see the benefit of holding this up any further.

[ahmetb]

@stevvooe I think we're a bit not on the same page about this: Ever since we saw the first design doc, the diagrams we've been looking at pretty much gave us the impression that what I proposed ("in case of cache-miss, download & stream at the same time") was going to be the strategy for handling cache misses. (instead of responding with 404/3xx and deferring to another registry)

That assumption made me also think if we host an in-datacenter Registry Mirror, any user hitting to this mirror can pull public Docker images (1) without hitting public internet (2) and paying any data transfer costs (because in-DC traffic is free on our platform).

In case of responding with 404/3xx (and delegating the client to download from somewhere else), this might not be what end users of the mirror might be interested at. Also by looking at the proposal above, I failed to find such details except the "If the remote issues a redirect[...]" part, which we should probably discuss further because it looks like folks on this thread thinks download&stream simultaneously looks like a good idea.

Agreed on “pro-active mirroring” should not be part of this at all. I just brought that up as a separate tool we will probably be building as Azure folks.

[stevvooe]

@ahmetalpbalkan I am confused: How does this proposal block the ability to download and stream at the same time? Compliant clients already have to follow redirects so I am failing to see what we've missed. Just because they may redirect, doesn't mean the server can't block and starting streaming a response instead. Put differently, on GET, a server may respond with the following:

1. Return a response with the data (200).
2. Redirect the client else where (307).
3. Indicate that data is unknown to the registry (404).

The specification was written this way and has always allowed this. Note that no part of this effort actually requires changes to the specification. Effectively, this support has always been a part of the V2 protocol. If a client handles these cases appropriately, this will always work.

At this point, I am confused about what changes would you actually like us to make. What do you want changed?

[ahmetb]

@stevvooe I'm assuming in the case of cache-miss the 2. Redirect the client else where (307). will happen. This behavior will most likely cause (1) extra slowness, (2) hitting public internet (not always desirable) and (3) bandwidth costs to the end user of the Registry Mirror (also not desirable).

I probably misunderstood the spec on the Google Docs and for some reason I assumed in case of cache-miss, we can just do "fetch & stream simultaneously" on the mirror and cache miss would never return 3. Indicate that data is unknown to the registry (404) by the mirror.

Having "fetch & stream simultaneously" would also make the cache-misses transparent to the end-client of the registry mirror. I would like to propose this part to be changed (@bobrik and @mattmoor are showing interest to this feature as well).

Also, did you quote these 3 behaviors from somewhere? It may as well be the case I don't have the link for a spec document and therefore I might be missing some context here. It still is not clear to me how the registry mirror would behave in the case of a cache-miss.

[mattmoor]

@stevvooe My interest is a separation of concerns. I would love to see a client-side spec / implementation independent of a registry spec / implementation.

I don't think it is hard to implement strategy-agnostic client-side logic, even if you have a specific implementation in mind for docker/distribution.

In particular, I'd love to see the client-side support pulling (a subset of) blobs through the mirror, including when the manifest 404s.

As I said, the design of the v1 mirroring code (in the client) mandated a pull-through implementation to be remotely usable, I'd like to avoid this in the v2 mirroring code.

My primary concern is the changes intended for the client, not the changes intended for the registry. I apologize this wasn't clearer, as I'm aware of only this one spec that covers both client / server.

[stevvooe]

@ahmetalpbalkan These 3 behaviors come from http. An http server has some control over what a client may do and these are the control points (I omitted 304 for brevity above). What a client does with those behaviors affects the caching behavior. What behavior is missing from that list?

I am not sure what you mean by "the Google Doc". I know there is a partner doc floating around but this isn't necessarily related to that. This is a component of that but not a complete solution.

@mattmoor What aspects of this current proposal mandate a pull-through implementation for the current or future mirroring features? Bear in mind, there will be a lot of work in the client to fully leverage any kind of caching. The 1.7 support is pretty narrow, since we had to work around what was already there from v1.

Again, what specific changes would you like to see in this proposal? Please identify exactly what is you want different.

[mattmoor]

@stevvooe my main interest is in changes to the client-side support, so if this proposal is solely for a v2 registry mirror that complies with the v1 mirroring limitations (which as we've discussed mandates pull-through), then the only thing I'd change is to call that out explicitly.

It certainly wasn't obvious to me that this was the case from this proposal or our discussion last week (I thought this was a client/server overhaul), but it seems that's not the case:

The 1.7 support is pretty narrow, since we had to work around what was already there from v1.

That said, when the client work starts, I'd appreciate if you'd loop me in:

Bear in mind, there will be a lot of work in the client to fully leverage any kind of caching

thanks.

[RichardScothern]

Merged in #779

[tobegit3hub]

Great job @RichardScothern 👍

[thaJeztah]

Nice!!

[BenHall]

Very nice to see this coming together. Are there any images built yet?

[dmp42]

@BenHall there is a 2.1.0-rc.0 over here: https://registry.hub.docker.com/u/distribution/registry/tags/manage/