Add pull through cache ability to the Registry. #779

RichardScothern · 2015-07-29T19:32:19Z

Add pull through cache ability to the Registry.

Create a Repository middleware which delegates storage to a proxyBlobStore
and proxyManifestStore. These stores will pull through data if not present
locally.

Create a Registry middleware which runs a scheduler to remove
expired content.

The scheduler runs as a background goroutine. When a blob
or manifest is pulled through from the remote registry,
an entry is added to the scheduler with a TTL. When the
TTL expires the scheduler calls a pre-specified function
to remove the fetched resource.

Add token authentication to the registry middleware.
Get a token at startup and preload the credential store
with the username and password supplied in the config file.

Signed-off-by: Richard Scothern richard.scothern@gmail.com

stevvooe · 2015-07-30T21:57:50Z

cmd/registry/config-cache.yml

+          disabled: true
+middleware:
+  repository:
+    - name: proxy


Does one have to configure both middlewares for this to work? I'm worried people will omit one or other or not even understand this. Can we make it so that the repository portion is not required?

Reduced to a single middleware configuration.

ahmetb · 2015-07-31T21:59:20Z

docs/mirror.md

+For example, if your mirror is serving on http://10.0.0.2:5000, you would run:
+
+```
+docker --registry-mirror=http://10.0.0.2:5000 -d


Hmm, wouldn't the docker daemon complain about this example address as it is not https?

It would. I'll change the example to https (and link the certs documentation)

RichardScothern · 2015-08-04T21:19:28Z

Now 100% unmiddlewared, @stevvooe

stevvooe · 2015-08-04T21:51:57Z

LGTM

aaronlehmann · 2015-08-04T22:46:24Z

registry/proxy/proxyblobstore.go

+	scheduler   *scheduler.TTLExpirationScheduler
+}
+
+var _ distribution.BlobStore = proxyBlobStore{}


What does this line do?

It asserts that proxyBlobStore implements the distribution.BlobStore
interface.

On Tue, Aug 4, 2015 at 3:46 PM, Aaron Lehmann notifications@github.com
wrote:

In registry/proxy/proxyblobstore.go
#779 (comment):

"github.com/docker/distribution"

"github.com/docker/distribution/context"

"github.com/docker/distribution/digest"

"github.com/docker/distribution/registry/proxy/scheduler"
+)

+// todo(richardscothern): from cache control header or config file
+const blobTTL = time.Duration(24 * 7 * time.Hour)
+
+type proxyBlobStore struct {

localStore distribution.BlobStore

remoteStore distribution.BlobService

scheduler *scheduler.TTLExpirationScheduler
+}

+var _ distribution.BlobStore = proxyBlobStore{}

What does this line do?

—
Reply to this email directly or view it on GitHub
https://github.com/docker/distribution/pull/779/files#r36251728.

aaronlehmann · 2015-08-04T22:59:36Z

LGTM, but note minor comments above.

…figured with a new `proxy` section in the configuration file. Create a new registry type which delegates storage to a proxyBlobStore and proxyManifestStore. These stores will pull through data if not present locally. proxyBlobStore takes care not to write duplicate data to disk. Add a scheduler to cleanup expired content. The scheduler runs as a background goroutine. When a blob or manifest is pulled through from the remote registry, an entry is added to the scheduler with a TTL. When the TTL expires the scheduler calls a pre-specified function to remove the fetched resource. Add token authentication to the registry middleware. Get a token at startup and preload the credential store with the username and password supplied in the config file. Allow resumable digest functionality to be disabled at runtime and disable it when the registry is a pull through cache. Signed-off-by: Richard Scothern <richard.scothern@gmail.com>

Add pull through cache ability to the Registry.

dalanlan · 2015-08-05T01:49:40Z

I was wondering how to set up a mirror registry for a private registry (gcr.io, for example).
Should we configure it as cmd/registry/config-cache.yml?
I do believe an example enclosed in the doc would help much:)

dmp42 · 2015-08-05T07:01:44Z

@dalanlan this is still experimental - mirroring of private registries is not yet a supported scenario (by docker itself).

dalanlan · 2015-08-05T07:25:31Z

Ic. Many thanks ;-)

dalanlan · 2015-08-05T07:26:51Z

Any pr i can trace though?

dmp42 · 2015-08-06T01:15:13Z

@dalanlan do you mean #779 ?

dalanlan · 2015-08-06T02:07:26Z

Hmm, as specified in #459

Unlike the V1 registry mirroring feature, a V2 registry mirror can proxy to a private registry.

I assume it's a coming feature. And since #779 is merged, i was wondering whether there is any other pr to trace.

dalanlan · 2015-08-06T02:21:03Z

Or this feature has been fully finished by #779, remaining experimental though. If that's the case, any new milestone ( will not be Registry/2.1 anymore ) in plan? :)

mbentley · 2015-08-06T03:18:08Z

Awesome, so far the pull through caching is working well for me on docker engine 1.8.0-rc2. I'll put more load on it tomorrow 👍

dmp42 · 2015-08-06T17:23:44Z

@dalanlan it's a bit early for registry 2.2.

For now, the docker engine itself only support Hub mirroring (though the registry side of it is not limited).

About the "experimental" status: we feel mirroring needs to be fire-tested before calling it "stable".

lvlv · 2015-08-12T03:12:36Z

@matt @olivier Do we have some instructions to set this up? It'll be super
useful for me. :-)
Thanks

On Fri, Aug 7, 2015 at 1:23 AM, Olivier Gambier notifications@github.com
wrote:

@dalanlan https://github.com/dalanlan it's a bit early for registry 2.2.

For now, the docker engine itself only support Hub mirroring (though the
registry side of it is not limited).

About the "experimental" status: we feel mirroring needs to be fire-tested
before calling it "stable".

—
Reply to this email directly or view it on GitHub
#779 (comment).

mbentley · 2015-08-12T11:01:04Z

@lvlv - I just did the following:

docker run -d --restart=always -p 5000:5000 --name v2-mirror \
  -v /data/v2-mirror:/var/lib/registry \
  distribution/registry:2.1.0-rc.0 \
  /var/lib/registry/config.mirror.yml

I have a config.mirror.yml file in my /data/v2-mirror directory on my host to provide my customized configuration that enables the mirror (and removes any other features I am currently not using). My config looks like this:

version: 0.1
log:
  level: info
storage:
    filesystem:
        rootdirectory: /var/lib/registry
http:
    addr: :5000
proxy:
  remoteurl: https://registry-1.docker.io

stevvooe · 2015-08-12T17:40:56Z

Please stop commenting on merged issues. This is not a support channel.

GordonTheTurtle added group/distribution labels Jul 29, 2015

thaJeztah mentioned this pull request Jul 29, 2015

WIP: Registry pull through cache #610

Closed

dmp42 assigned RichardScothern Jul 29, 2015

dmp42 added this to the Registry/2.1 milestone Jul 29, 2015

RichardScothern force-pushed the pull-through-cache branch from 1bae712 to e39b171 Compare July 29, 2015 20:47

RichardScothern added the status/2-code-review label Jul 29, 2015

thaJeztah mentioned this pull request Jul 29, 2015

Caching repository jpetazzo/dind#65

Closed

RichardScothern force-pushed the pull-through-cache branch 2 times, most recently from bf39258 to 375afb0 Compare July 30, 2015 00:45

RichardScothern mentioned this pull request Jul 30, 2015

Support for pull-through cached mirroring #19

Closed

RichardScothern force-pushed the pull-through-cache branch from 375afb0 to de5d7ca Compare July 30, 2015 17:50

stevvooe reviewed Jul 30, 2015
View reviewed changes

dmp42 removed the status/0-triage label Jul 31, 2015

RichardScothern force-pushed the pull-through-cache branch from de5d7ca to 4d58472 Compare July 31, 2015 18:10

ahmetb reviewed Jul 31, 2015
View reviewed changes

RichardScothern force-pushed the pull-through-cache branch 2 times, most recently from abb5f3b to ad39ef9 Compare August 4, 2015 20:16

aaronlehmann reviewed Aug 4, 2015
View reviewed changes

RichardScothern force-pushed the pull-through-cache branch from ad39ef9 to 94935f3 Compare August 4, 2015 23:12

stevvooe added a commit that referenced this pull request Aug 5, 2015

Merge pull request #779 from RichardScothern/pull-through-cache

a0c6337

Add pull through cache ability to the Registry.

stevvooe merged commit a0c6337 into distribution:master Aug 5, 2015

stevvooe added feature and removed group/distribution status/2-code-review labels Aug 5, 2015

RichardScothern mentioned this pull request Aug 5, 2015

Registry V2 mirror proposal #459

Closed

12 tasks

distribution locked and limited conversation to collaborators Aug 12, 2015

RichardScothern deleted the pull-through-cache branch October 30, 2015 21:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pull through cache ability to the Registry. #779

Add pull through cache ability to the Registry. #779

RichardScothern commented Jul 29, 2015

stevvooe Jul 30, 2015

RichardScothern Aug 1, 2015

ahmetb Jul 31, 2015

RichardScothern Jul 31, 2015

RichardScothern commented Aug 4, 2015

stevvooe commented Aug 4, 2015

aaronlehmann Aug 4, 2015

RichardScothern Aug 4, 2015

aaronlehmann commented Aug 4, 2015

dalanlan commented Aug 5, 2015

dmp42 commented Aug 5, 2015

dalanlan commented Aug 5, 2015

dalanlan commented Aug 5, 2015

dmp42 commented Aug 6, 2015

dalanlan commented Aug 6, 2015

dalanlan commented Aug 6, 2015

mbentley commented Aug 6, 2015

dmp42 commented Aug 6, 2015

lvlv commented Aug 12, 2015

mbentley commented Aug 12, 2015

stevvooe commented Aug 12, 2015

Add pull through cache ability to the Registry. #779

Add pull through cache ability to the Registry. #779

Conversation

RichardScothern commented Jul 29, 2015

stevvooe Jul 30, 2015

Choose a reason for hiding this comment

RichardScothern Aug 1, 2015

Choose a reason for hiding this comment

ahmetb Jul 31, 2015

Choose a reason for hiding this comment

RichardScothern Jul 31, 2015

Choose a reason for hiding this comment

RichardScothern commented Aug 4, 2015

stevvooe commented Aug 4, 2015

aaronlehmann Aug 4, 2015

Choose a reason for hiding this comment

RichardScothern Aug 4, 2015

Choose a reason for hiding this comment

aaronlehmann commented Aug 4, 2015

dalanlan commented Aug 5, 2015

dmp42 commented Aug 5, 2015

dalanlan commented Aug 5, 2015

dalanlan commented Aug 5, 2015

dmp42 commented Aug 6, 2015

dalanlan commented Aug 6, 2015

dalanlan commented Aug 6, 2015

mbentley commented Aug 6, 2015

dmp42 commented Aug 6, 2015

lvlv commented Aug 12, 2015

mbentley commented Aug 12, 2015

stevvooe commented Aug 12, 2015