We have a continuous integration pipeline on circleci that does the following:
I ran the build twice in a row, and I see a lot of crossover in the hash of the layers being pushed. Yet rather than "Image already exists" I see "Image successfully pushed".
Here's the output of build 1's docker push, and here's build 2
If you diff those two files you'll see that only 2 layers differ in each build:
< ca44fed88be6: Buffering to Disk
< ca44fed88be6: Image successfully pushed
< 5dbd19bfac8a: Buffering to Disk
< 5dbd19bfac8a: Image successfully pushed
> 9136b10cfb72: Buffering to Disk
> 9136b10cfb72: Image successfully pushed
> 0388311b6857: Buffering to Disk
> 0388311b6857: Image successfully pushed
So why is it that all the images have to re-push every time?
It seems they have their own fork of docker they're using in the env...
$ docker -v
Docker version 1.8.2-circleci, build a8b52f5
$ docker info
Storage Driver: btrfs
Execution Driver: lxc-1.0.8
Logging Driver: json-file
Kernel Version: 3.13.0-76-generic
Operating System: Ubuntu precise (12.04.5 LTS) (containerized)
Total Memory: 240.2 GiB
Debug mode (server): true
File Descriptors: 17
System Time: 2016-01-22T12:32:38.134432556Z
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled```
running docker push repo/myimage:mytag2 twice in a row gives me the desired Image already exists for all layers.
docker push repo/myimage:mytag2
Image already exists
in addition, running:
$ docker tag repo/myimage:mytag2 repo/myimage:mytag3
$ docker push repo/myimage:mytag3
also gives me Image already exists for all layers
I think it's the save/load that's causing problems here. Docker caches hashes of the layers in /var/lib/docker, but without that cache, it has to push each layer from scratch. The cache is not carried along with tarballs produced by docker save. I believe CircleCI wipes the /var/lib/docker directory between runs, so you can't benefit from this cache.
Ideally CircleCI would offer a way to preserve the /var/lib/docker directory instead of relying on the save/load hack. However, it may be possible to work around this by backing up /var/lib/docker/*/checksum (for versions < 1.10) or /var/lib/docker/image/<graph driver>/distribution (for versions >= 1.10).
so assuming I did this right, that doesn't seem to work
I added /var/lib/docker/image/btrfs/distribution to the cache directory (assuming btrfs is the right graph driver based on "Storage Driver" in docker info) and I still get full pushed T_T
ah. it seems what's going on is that the cache is saved after the dependencies step, which is obviously before deployment (docker push) runs...trying to come up with a good solution here - i'll keep updating the thread for anyone else who runs into this issue but as it's obviously not a docker issue i'll close it
For anyone else who ends up here:
There is a "better" solution needed to be implemented on the CircleCI side in terms of the cached dirs and stuff. However one issue is that CircleCI is on 1.8.x. The 1.9.1 upgrade greatly improves buffering for pushes. You can enable 1.9.1, see here: https://discuss.circleci.com/t/docker-1-9-1-is-available/1009.
This improved my buffer/push speed in Circle significantly.