Subsequent COPY instructions re-adds all files in every layer instead of only the files that have changed

[motin]

Under certain circumstances, subsequent COPY instructions re-adds all files in every layer instead of only the files that have changed.

ISSUE RENAMED: The original suggestion was to add a SYNC instruction, but the COPY instruction should already cover the use cases for which the SYNC instructions was intended. The original post was as follows:

Suggestion:
The SYNC instruction compares and and performs the necessary changes for to become identical to .

This would alleviate a lot of pain present when building images that contain application source code that changes incrementally and often, since the whole source code needs to be COPIED (COPY . /app) to the image again and again, leading to longer deploy cycles and tons of unnecessary data pushed to and stored in the registries.

In extreme cases, the source code directory can reach up to gigabytes in size, and in order to publish a some smaller changes in a few hundred files, all gigabytes needs to be pushed and subsequently pulled by those who wish to get access to the changes, or the servers that are to serve the new source code.

A SYNC command would instead allow for new image layers to include only the files that actually changed in comparison to the existing image contents, which probably won't amount to more than a few hundred kilobytes in most cases.

(Optionally, an ability to access the build context from the RUN instruction would make this particular instruction unnecessary by allowing for instance rsync to compare build context against image contents)

The only currently feasible workaround today seems to be to use rsync to analyze the differences between two images and then use the changelog output to craft a tar-file containing the relevant changes.

[phemmer]

Docker already does this. When docker generates a new layer, it calculates the difference. It doesn't store the entire filesystem.

This can be demonstrated in the following example:

# ll
total 97688
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 0.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 1.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 2.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 3.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 4.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 5.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 6.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 7.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:37 8.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:37 9.data
-rw-r--r-- 1 phemmer adm       22 2016/04/12-08:21:35 Dockerfile
-rw-r--r-- 1 phemmer adm       19 2016/04/12-08:27:47 Dockerfile.update

# cat Dockerfile
FROM busybox
COPY . /

# cat Dockerfile.update 
FROM test
COPY . /

# docker build -t test .
Sending build context to Docker daemon   100 MB
Step 0 : FROM busybox
 ---> 8c2e06607696
Step 1 : COPY . /
 ---> 8708bc7d48c9
Removing intermediate container c61f255a6bd1
Successfully built 8708bc7d48c9

# echo foo >> 0.data

# docker build -t test -f Dockerfile.update .                                                         
Sending build context to Docker daemon   100 MB
Step 0 : FROM test
 ---> 8708bc7d48c9
Step 1 : COPY . /
 ---> da36496146d8
Removing intermediate container 8cf413dc4621
Successfully built da36496146d8

# docker history test
IMAGE               CREATED              CREATED BY                                      SIZE                COMMENT
da36496146d8        21 seconds ago       /bin/sh -c #(nop) COPY dir:ec5ab01cc732f96db7   10 MB               
8708bc7d48c9        About a minute ago   /bin/sh -c #(nop) COPY dir:5647d3b25c72c1acab   100 MB              
8c2e06607696        12 months ago        /bin/sh -c #(nop) CMD ["/bin/sh"]               0 B                 
6ce2e90b0bc7        12 months ago        /bin/sh -c #(nop) ADD file:8cf517d90fe79547c4   2.43 MB             
cf2616975b4a        12 months ago        /bin/sh -c #(nop) MAINTAINER J�r�me Petazzo     0 B                 

Notice how the size of the last layer (first in the list since docker outputs reversed history) is 10mb, which is the size of the file we changed. All the other 90mb worth of files were unchanged, and thus not added to the layer.

[motin]

@phemmer Whoa, great news! But in what Docker version was this fixed? I am using 1.10.3 on OSX (latest available from https://www.docker.com/products/docker-toolbox) and getting:

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
6512838f1036        4 seconds ago       /bin/sh -c #(nop) COPY dir:7d72c14da84a2496de   100 MB
b2d4c3b36a37        20 seconds ago      /bin/sh -c #(nop) COPY dir:25e5b0c3d4bd31c7c6   100 MB
47bcc53f74dc        3 weeks ago         /bin/sh -c #(nop) CMD ["sh"]                    0 B
<missing>           3 weeks ago         /bin/sh -c #(nop) ADD file:47ca6e777c36a4cfff   1.113 MB

Notice how the size of the last layer (first in the list since docker outputs reversed history) is 100mb, meaning that all files were re-added in the last layer.

[thaJeztah]

@motin the difference is that in the example @phemmer gave, he split the image up into a "base" image (using Dockerfile) and an image that extends the image (Dockerfile.update). If you only rebuild the second one, docker compares the change with the layers above it (which are in the base-image), and only adds files that are changed.

The building is still done on the daemon, so docker will upload the whole build-context to the daemon on each build (#9553 is a proposal to make this smarter)

[motin]

@thaJeztah I too split up the image in the exact same way. Actually, I re-ran @phemmer's example from top to bottom and got the 100MB-sized layer outcome. Maybe there is some bug causing these differences in outcome? On what system and with what Docker version does it work? It sure does not work on 1.10.3 on OSX.

[thaJeztah]

@motin oh! I think there's a typo in in @phemmer's example; he's using the same name twice (both in the first and second build). change -t test to something else in the second build, and it should work

[thaJeztah]

hm, although, that shouldn't matter the first time you run it (it should just untag the image)

[motin]

@thaJeztah Thanks, it did not change the outcome however.

I set up a gist and the following one-liner to replicate the issue locally:

git clone https://gist.github.com/cca880c647263eb5e98d9f1e0d60a3c5.git replicate-docker-issue-21950 && bash replicate-docker-issue-21950/replicate-docker-issue-21950.sh

Here is my output from the one-liner

[thaJeztah]

hm, looking into this now; it seems that for some reason it's indeed including all files again in the last layer, not just the changed file

[motin]

@thaJeztah Thanks, I renamed the issue to better reflect the underlying issue.

[thaJeztah]

Yes, these were my steps to reproduce;

Dockerfile.base;

FROM scratch
COPY . /data

Dockerfile

FROM test
COPY . /data

mkfile 10m data.0
docker build -t test -f Dockerfile.base .

mkfile 10m data.1
docker build -t test .

Then save the image (docker save -o test.tar test ), and extracting the image, and the layers, I get;

# snip #
├── 81650965dc14fed041e07f15b40af4e662d422f2914365aa4a2d2f704ee729a8
│   ├── VERSION
│   ├── data
│   │   ├── Dockerfile
│   │   ├── Dockerfile.base
│   │   ├── data.0
│   │   └── data.1
│   ├── json
├── 82963b397b3e53fc9cecbb81f45c81591fa1fe48c8c71fc5b3013470636c5122
│   ├── VERSION
│   ├── data
│   │   ├── Dockerfile
│   │   ├── Dockerfile.base
│   │   └── data.0
│   ├── json

So data.0 seems to be embedded twice in the image, in each layer that added it.

(testing on 1.11.0-rc4)

[phemmer]

@motin oh! I think there's a typo in in @phemmer's example; he's using the same name twice (both in the first and second build). change -t test to something else in the second build, and it should work

No typo. That was deliberate (and works fine).

[motin]

@thaJeztah What is your output from the one-liner above and one what system are you using Docker?