docker registry proxy for travis builders

[KostyaSha]

Hi, do you have any local to builders docker-registry proxy?
https://blog.docker.com/2015/10/registry-proxy-cache-docker-open-source/

[janisozaur]

This would be a really good addition to Travis CI. Many of docker images are based on common images, like the official Ubuntu ones, adding a local cache would certainly speed things up.

I'm looking forward to seeing it implemented.

[edmorley]

In case it saves anyone else time who is thinking of testing the same, I tried using Google's Container Registry mirror of Docker Hub to see if it would improve pull performance (given that Travis' sudo-enabled jobs run on GCE):
https://cloud.google.com/container-registry/docs/using-dockerhub-mirroring
kubernetes/kubernetes#25841

Eg using:

install:
  - echo 'DOCKER_OPTS="$DOCKER_OPTS --debug --registry-mirror=https://us-mirror.gcr.io"' | sudo tee -a /etc/default/docker > /dev/null
  - sudo service docker restart
  - docker system info
script:
  - docker --log-level debug pull ubuntu:latest
  - docker --log-level debug pull ubuntu:16.04
  - docker --log-level debug pull heroku/heroku:16
  - sudo cat /var/log/upstart/docker.log

Example run:
https://travis-ci.org/edmorley/travis-testing/builds/252211908

Log output of note:

time="2017-07-10T23:30:16.525306602Z" level=debug msg="Trying to pull ubuntu from https://us-mirror.gcr.io/ v2" 
time="2017-07-10T23:30:16.602624904Z" level=info msg="Attempting next endpoint for pull after error: manifest unknown: Failed to fetch \"16.04\" from request \"/v2/library/ubuntu/manifests/16.04\"." 
time="2017-07-10T23:30:16.602682070Z" level=debug msg="Trying to pull ubuntu from https://registry-1.docker.io v2" 

Thus sadly:
(a) unfortunately the mirror seems to only support official images and even then only of tag latest (see here),
(b) even for the ubuntu:latest image in the example above, the pull time is about the same as with no mirror (I tried both mirror.gcr.io and us-mirror.gcr.io).

...so my idea of updating /etc/default/docker in the base GCE image (defined here) to point at this mirror isn't going to help.

It's possible a Travis registry mirror hosted in the same GCE zone as the sudo-enabled jobs themselves might still be faster of course.

[ktomk]

I suggest to give this perhaps a bump. If Travis has the space and the images were coming from a place nearby (I think for Composer (PHP) there are similar speedups but Packagist packages are never that fast on Travis in my observation). Perhaps it's possible to have something similar for the Docker Hub registry?

I see other Travis issues where devs try around with image caches but I think that's counter-productive and it also won't scale well by providing little benefit b/c of S3 speed.

[taqtiqa-mark]

In issue #9384 I posed the question "Does Travis-CI already have a Docker registry mirror configured as a "pull through cache"?" It seems the answer is no.
Are there plans to?

It appears that the issue can be more severe than just time to build an image.

Issue #9384 identifies situations where the builds fail because some bandwidth limit is hit.

Does anyone have experience with builds downloading and producing multiple artifacts >4GB?

[kfox1111]

Seems like this was partially implemented, as its causing problems with some builds. See here for details:
moby/moby#39120