Docker hub image cache doesn't seem to be working #19583

Closed
jtmarmon opened this Issue Jan 22, 2016 · 6 comments

Projects

None yet

3 participants

@jtmarmon

We have a continuous integration pipeline on circleci that does the following:

  1. Loads repo/image:mytag1 from the cache directory to be able to use cached layers
  2. Builds a new version: docker build -t repoimage:mytag2
  3. Saves the new version to the cache directory with docker save
  4. Runs tests
  5. Pushes to docker hub: docker push repo/image:mytag2
    The problem is with step 5. The push step takes 5 minutes every time. If I understand it correctly, docker hub is meant to cache layers so we don't have to re-push things like the base image and dependencies if they are not updated.

I ran the build twice in a row, and I see a lot of crossover in the hash of the layers being pushed. Yet rather than "Image already exists" I see "Image successfully pushed".

Here's the output of build 1's docker push, and here's build 2

If you diff those two files you'll see that only 2 layers differ in each build:

< ca44fed88be6: Buffering to Disk
< ca44fed88be6: Image successfully pushed
< 5dbd19bfac8a: Buffering to Disk
< 5dbd19bfac8a: Image successfully pushed

---
> 9136b10cfb72: Buffering to Disk
> 9136b10cfb72: Image successfully pushed
> 0388311b6857: Buffering to Disk
> 0388311b6857: Image successfully pushed

So why is it that all the images have to re-push every time?

Environment info:

It seems they have their own fork of docker they're using in the env...

$ docker -v
Docker version 1.8.2-circleci, build a8b52f5
$ docker info
Containers: 3
Images: 47
Storage Driver: btrfs
Execution Driver: lxc-1.0.8
Logging Driver: json-file
Kernel Version: 3.13.0-76-generic
Operating System: Ubuntu precise (12.04.5 LTS) (containerized)
CPUs: 2
Total Memory: 240.2 GiB
Name: box1289
ID: IDME:TMNA:PWC4:OQ4A:LMJR:K7RC:NUJ6:LDSZ:7JTN:I4B3:VO5G:HMSB
Debug mode (server): true
File Descriptors: 17
Goroutines: 28
System Time: 2016-01-22T12:32:38.134432556Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled```
@jtmarmon

more info:

running docker push repo/myimage:mytag2 twice in a row gives me the desired Image already exists for all layers.

in addition, running:

$ docker tag repo/myimage:mytag2 repo/myimage:mytag3
$ docker push repo/myimage:mytag3

also gives me Image already exists for all layers

@aaronlehmann
Contributor

I think it's the save/load that's causing problems here. Docker caches hashes of the layers in /var/lib/docker, but without that cache, it has to push each layer from scratch. The cache is not carried along with tarballs produced by docker save. I believe CircleCI wipes the /var/lib/docker directory between runs, so you can't benefit from this cache.

Ideally CircleCI would offer a way to preserve the /var/lib/docker directory instead of relying on the save/load hack. However, it may be possible to work around this by backing up /var/lib/docker/*/checksum (for versions < 1.10) or /var/lib/docker/image/<graph driver>/distribution (for versions >= 1.10).

@jtmarmon

so assuming I did this right, that doesn't seem to work

I added /var/lib/docker/image/btrfs/distribution to the cache directory (assuming btrfs is the right graph driver based on "Storage Driver" in docker info) and I still get full pushed T_T

@jtmarmon

ah. it seems what's going on is that the cache is saved after the dependencies step, which is obviously before deployment (docker push) runs...trying to come up with a good solution here - i'll keep updating the thread for anyone else who runs into this issue but as it's obviously not a docker issue i'll close it

@jtmarmon jtmarmon closed this Jan 23, 2016
@alph486
alph486 commented Apr 19, 2016

For anyone else who ends up here:

There is a "better" solution needed to be implemented on the CircleCI side in terms of the cached dirs and stuff. However one issue is that CircleCI is on 1.8.x. The 1.9.1 upgrade greatly improves buffering for pushes. You can enable 1.9.1, see here: https://discuss.circleci.com/t/docker-1-9-1-is-available/1009.

This improved my buffer/push speed in Circle significantly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment