Subsequent COPY instructions re-adds all files in every layer instead of only the files that have changed

[motin]

Under certain circumstances, subsequent COPY instructions re-adds all files in every layer instead of only the files that have changed.

ISSUE RENAMED: The original suggestion was to add a SYNC instruction, but the COPY instruction should already cover the use cases for which the SYNC instructions was intended. The original post was as follows:

Suggestion:
The SYNC instruction compares and and performs the necessary changes for to become identical to .

This would alleviate a lot of pain present when building images that contain application source code that changes incrementally and often, since the whole source code needs to be COPIED (COPY . /app) to the image again and again, leading to longer deploy cycles and tons of unnecessary data pushed to and stored in the registries.

In extreme cases, the source code directory can reach up to gigabytes in size, and in order to publish a some smaller changes in a few hundred files, all gigabytes needs to be pushed and subsequently pulled by those who wish to get access to the changes, or the servers that are to serve the new source code.

A SYNC command would instead allow for new image layers to include only the files that actually changed in comparison to the existing image contents, which probably won't amount to more than a few hundred kilobytes in most cases.

(Optionally, an ability to access the build context from the RUN instruction would make this particular instruction unnecessary by allowing for instance rsync to compare build context against image contents)

The only currently feasible workaround today seems to be to use rsync to analyze the differences between two images and then use the changelog output to craft a tar-file containing the relevant changes.

[phemmer]

Docker already does this. When docker generates a new layer, it calculates the difference. It doesn't store the entire filesystem.

This can be demonstrated in the following example:

# ll
total 97688
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 0.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 1.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 2.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 3.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 4.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 5.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 6.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:36 7.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:37 8.data
-rw-r--r-- 1 phemmer adm 10000000 2016/04/12-08:29:37 9.data
-rw-r--r-- 1 phemmer adm       22 2016/04/12-08:21:35 Dockerfile
-rw-r--r-- 1 phemmer adm       19 2016/04/12-08:27:47 Dockerfile.update

# cat Dockerfile
FROM busybox
COPY . /

# cat Dockerfile.update 
FROM test
COPY . /

# docker build -t test .
Sending build context to Docker daemon   100 MB
Step 0 : FROM busybox
 ---> 8c2e06607696
Step 1 : COPY . /
 ---> 8708bc7d48c9
Removing intermediate container c61f255a6bd1
Successfully built 8708bc7d48c9

# echo foo >> 0.data

# docker build -t test -f Dockerfile.update .                                                         
Sending build context to Docker daemon   100 MB
Step 0 : FROM test
 ---> 8708bc7d48c9
Step 1 : COPY . /
 ---> da36496146d8
Removing intermediate container 8cf413dc4621
Successfully built da36496146d8

# docker history test
IMAGE               CREATED              CREATED BY                                      SIZE                COMMENT
da36496146d8        21 seconds ago       /bin/sh -c #(nop) COPY dir:ec5ab01cc732f96db7   10 MB               
8708bc7d48c9        About a minute ago   /bin/sh -c #(nop) COPY dir:5647d3b25c72c1acab   100 MB              
8c2e06607696        12 months ago        /bin/sh -c #(nop) CMD ["/bin/sh"]               0 B                 
6ce2e90b0bc7        12 months ago        /bin/sh -c #(nop) ADD file:8cf517d90fe79547c4   2.43 MB             
cf2616975b4a        12 months ago        /bin/sh -c #(nop) MAINTAINER J�r�me Petazzo     0 B                 

Notice how the size of the last layer (first in the list since docker outputs reversed history) is 10mb, which is the size of the file we changed. All the other 90mb worth of files were unchanged, and thus not added to the layer.

[motin]

@phemmer Whoa, great news! But in what Docker version was this fixed? I am using 1.10.3 on OSX (latest available from https://www.docker.com/products/docker-toolbox) and getting:

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
6512838f1036        4 seconds ago       /bin/sh -c #(nop) COPY dir:7d72c14da84a2496de   100 MB
b2d4c3b36a37        20 seconds ago      /bin/sh -c #(nop) COPY dir:25e5b0c3d4bd31c7c6   100 MB
47bcc53f74dc        3 weeks ago         /bin/sh -c #(nop) CMD ["sh"]                    0 B
<missing>           3 weeks ago         /bin/sh -c #(nop) ADD file:47ca6e777c36a4cfff   1.113 MB

Notice how the size of the last layer (first in the list since docker outputs reversed history) is 100mb, meaning that all files were re-added in the last layer.

[thaJeztah]

@motin the difference is that in the example @phemmer gave, he split the image up into a "base" image (using Dockerfile) and an image that extends the image (Dockerfile.update). If you only rebuild the second one, docker compares the change with the layers above it (which are in the base-image), and only adds files that are changed.

The building is still done on the daemon, so docker will upload the whole build-context to the daemon on each build (#9553 is a proposal to make this smarter)

[motin]

@thaJeztah I too split up the image in the exact same way. Actually, I re-ran @phemmer's example from top to bottom and got the 100MB-sized layer outcome. Maybe there is some bug causing these differences in outcome? On what system and with what Docker version does it work? It sure does not work on 1.10.3 on OSX.

[thaJeztah]

@motin oh! I think there's a typo in in @phemmer's example; he's using the same name twice (both in the first and second build). change -t test to something else in the second build, and it should work

[thaJeztah]

hm, although, that shouldn't matter the first time you run it (it should just untag the image)

[motin]

@thaJeztah Thanks, it did not change the outcome however.

I set up a gist and the following one-liner to replicate the issue locally:

git clone https://gist.github.com/cca880c647263eb5e98d9f1e0d60a3c5.git replicate-docker-issue-21950 && bash replicate-docker-issue-21950/replicate-docker-issue-21950.sh

Here is my output from the one-liner

[thaJeztah]

hm, looking into this now; it seems that for some reason it's indeed including all files again in the last layer, not just the changed file

[motin]

@thaJeztah Thanks, I renamed the issue to better reflect the underlying issue.

[thaJeztah]

Yes, these were my steps to reproduce;

Dockerfile.base;

FROM scratch
COPY . /data

Dockerfile

FROM test
COPY . /data

mkfile 10m data.0
docker build -t test -f Dockerfile.base .

mkfile 10m data.1
docker build -t test .

Then save the image (docker save -o test.tar test ), and extracting the image, and the layers, I get;

# snip #
├── 81650965dc14fed041e07f15b40af4e662d422f2914365aa4a2d2f704ee729a8
│   ├── VERSION
│   ├── data
│   │   ├── Dockerfile
│   │   ├── Dockerfile.base
│   │   ├── data.0
│   │   └── data.1
│   ├── json
├── 82963b397b3e53fc9cecbb81f45c81591fa1fe48c8c71fc5b3013470636c5122
│   ├── VERSION
│   ├── data
│   │   ├── Dockerfile
│   │   ├── Dockerfile.base
│   │   └── data.0
│   ├── json

So data.0 seems to be embedded twice in the image, in each layer that added it.

(testing on 1.11.0-rc4)

[phemmer]

@motin oh! I think there's a typo in in @phemmer's example; he's using the same name twice (both in the first and second build). change -t test to something else in the second build, and it should work

No typo. That was deliberate (and works fine).

[motin]

@thaJeztah What is your output from the one-liner above and one what system are you using Docker?

[motin]

No typo. That was deliberate.

Thanks for the clarification. The gist has been updated to use the same tags as in your example: https://gist.github.com/motin/cca880c647263eb5e98d9f1e0d60a3c5

Still the same 100MB-sized layer outcome regardless of the tags used.

[phemmer]

To answer the question about what version I'm using, 1.8.3 (the underlying os is always linux, even on mac. Though the client is linux, just in case that for some unlikely reason makes a difference).

[motin]

To answer the question about what version I'm using, 1.8.3

@phemmer Thanks. I now ran the one-liner on Docker 1.8.3 running on Debian, however the issue remains. Apparently, this has nothing to do with Docker version or if it is running in a native Linux environment or in OSX.

What do you get if you run the one-liner on your box?

[thaJeztah]

@phemmer having the same on 1.9.1, now trying docker 1.8.3

[schmunk42]

I also ran the script on 1.7 (Debian), 1.10 (Ubuntu), 1.11-rc4 (Mac) on Mac, via SSH and a swarm, all with the 100 MB / 100 MB outcome.

[update]
All tests were executed on AUFS.

[justincormack]

Can replicate on aufs, but on overlay I get 10MB/100MB.

[phemmer]

I was beginning to wonder if this was specific to the storage driver. I'm using btrfs. (and I get the proper behavior)

[thaJeztah]

looks to be aufs yes

[schmunk42]

Slightly OT: But how to test other storage drivers (easily) - I looked through the docs and also tried to create machines with docker-machine but none of them with other storage drivers than aufs or devicemapper were working for me.

And (bonus question) what's the current state or plan about docker and it's preferred storage driver?

[justincormack]

@schmunk42 Docker for Mac can switch between aufs and overlay, although it is not yet documented I see. Generally those are easy to switch as they do not involve any reformatting, you can just change the /etc/docker/daemon.json config file or the startup arguments.

In terms of advice well there is @jfrazelle view here https://blog.jessfraz.com/post/the-brutally-honest-guide-to-docker-graphdrivers/ - it heavily depends on what base distro you are using what choice you make.

[schmunk42]

Confirmed to be working as mentioned (10 MB / 100 MB) on 1.11.0-rc5 with overlay.

[AkihiroSuda]

@schmunk42 This page is also great: https://github.com/docker/docker/blob/master/docs/userguide/storagedriver/selectadriver.md#future-proofing

It says AUFS and Devicemapper(direct-lvm) are "production-ready" status.

[jessfraz]

I would agree with AUFS but you've been warned about devicemapper

On Tue, Apr 12, 2016 at 9:04 AM, Akihiro Suda notifications@github.com
wrote:

@schmunk42 https://github.com/schmunk42 This page is also great:
https://github.com/docker/docker/blob/master/docs/userguide/storagedriver/selectadriver.md#future-proofing

It says AUFS and Devicemapper(direct-lvm) are "production-ready" status.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#21950 (comment)

Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3
pgp.mit.edu http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3

[jessfraz]

[tonistiigi]

This is happening because of the different methods storage drivers use to compute layer differences. Everything but aufs uses NaiveDiffDriver that basically means that on every commit new and parent directories are compared based on file sizes and modified times. Aufs uses the native filesystem features and the new layer is the copy-up folder. Afaik builder itself doesn't optimize this case atm so all files are always copied, only in case of the naive driver some files are ignored later. Changing the diff method for aufs isn't a solution as it has many benefits(both performance and accuracy). There are some plans to change overlay to a similar method also.

[vitaliitylyk]

3 years, is there any update on this? :)

[tomwillfixit]

Using dive to debug why some images are so large and came across this issue.

In the Dockerfile I'm copying from a base image and it results in multiple copies of the same file.

COPY --from=base /usr/local/lib /usr/local/lib

If anyone has a workaround to skip duplicate files during the COPY, that'd be great. Thanks.

[thaJeztah]

In the Dockerfile I'm copying from a base image and it results in multiple copies of the same file.

If the target stage already has those files then depending on the storage driver, those files may be copied over it (even if they would possibly be the same). More details about that in #35280

[corydolphin]

To join in on the fun, I'm running into this issue on aufs and overlay2fs as well. Is there any more direct issue with the storage implementations that we should move the discussion to? I expect many other users are hitting this issue, but don't realize the bloated size of their images.

[h0jeZvgoxFepBQ2C]

I'm having this issue on OSX Catalina latest version.. Guess this issue wont be fixed if its already open serveral years now 🙈 😢

[h0jeZvgoxFepBQ2C]

Is there no workaround for this? We are pushing 200mb each time, even if we only change a 1 bit file.. 🙄

[thaJeztah]

@h0jeZvgoxFepBQ2C do you have BuildKit enabled (DOCKER_BUILDKIT=1)? Are the "repeated COPY" steps in a single build, or are you using a pre-existing image (pulled from a registry), and COPY files that were already in the image?

[electrofelix]

@h0jeZvgoxFepBQ2C our workaround is to use the same base stage, in the one that we are modifying and will copy from to the final layer, perform a RUN find <path-that-will-be-copied> -type f > /files-to-delete.txt, make the required modifications, and then add a RUN cat /files-to-delete.txt | xargs rm -f. The subsequent copy statement of COPY --from=<stage> <path-that-will-be-copied> <path-that-will-be-copied> will then only contain the files added.

Something like the following (note we're using buildkit to do some caching of packages)

# syntax = docker/dockerfile:1.0-experimental
FROM alpine:3.12.3 as py-base

RUN --mount=type=cache,target=/var/cache/apk apk add --progress \
        ca-certificates \
        git \
        python3 \
        openssh \
        openssl \
        sshpass \
    ;

FROM py-base as ansible-build
RUN --mount=type=cache,target=/var/cache/apk apk add --update \
            python3-dev \
            libffi-dev \
            openssl-dev \
            build-base \
    ;

RUN find /usr/bin/ /usr/lib/ -type f > /files-to-delete.txt

COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache \
    HOME=/root pip3 install --upgrade pip \
    && HOME=/root pip3 install -r requirements.txt \
    ;

RUN cat /files-to-delete.txt | xargs rm -f

FROM py-base

COPY --from=ansible-build /usr/lib /usr/bin /usr

This is not ideal in that we're installing packages in the second stage that might end up being different versions compared to the libraries due to caching of layers, but in practice we haven't been bitten by it yet.

I subsequently found a better way in that the python install of ansible is relocatable so I can instead install using HOME=/root pip3 install -r requirements.txt --prefix /usr/local and the copy becomes COPY --from=ansible-build /usr/local/* /usr/ avoiding the need to delete files to prevent duplicate copies.

But the idea should work in the meantime as a way of brute forcing ignore existing files.

@thaJeztah to confirm we bumped into this about 2 months ago, and have buildkit enabled by default for our builds. My storage is:

 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true

If there is anything you'd like me to test to add more data, let me know.

[blackwood821]

Is this bug ever going to be fixed?

[ia-davidpichler]

I'm getting messages from the docker daemon that the overlay driver is deprecated and will be removed. Is there a different driver that has the correct behavior of not copying unchanged files?

[t0rrant]

I just noticed this issue as well.
Changing the storage driver from overlay2 to overlay seems to make COPY behave as expected, but overlay is deprecated?

The legacy overlay driver was used for kernels that did not support the “multiple-lowerdir” feature required for overlay2 All currently supported Linux distributions now provide support for this, and it is therefore deprecated.

-- from https://docs.docker.com/storage/storagedriver/select-storage-driver/

[h0jeZvgoxFepBQ2C]

We are still having this problem... For me on OSX it works fine now with the depreacted overlay driver.

But my team members with Windows / Ubuntu WSL2 still have this problem, even with overlay driver... Any hints?

[h0jeZvgoxFepBQ2C]

This is still an issue and it's really a pain to always send 600MB even if one one small text file changes. I cannot understand how overlay gets deprecated, even though this bug still exists.

Today I switched back to overlay2 again and the issue is still persisting.

@thaJeztah Our workflow with the old driver, which works fine, was to build a base image with all files included, and another image which is based on the base image, but adds all files again. With verlay only small layer changes are happening, overlay2 is behaving like all files are completely new/changed.

[Withy14]

I have tested one of the scenarious above. The drivers I used: vfs, btrfs, overlay, devicemppper, fuse-overalyfs,, zfs are behaving the same way. They do not re-add all files. Only Overlay2 is having this issue. So I think that if yu do not want to rely on overlay (because it is deprecated despite the fact that overlay2 cannot be considered as a replacement I would suggest to try the btrfs or zfs) or "throw" away docker for building and use directly buildkit.

[vafgoettlich]

@Withy14 what do you mean with "use directly builtkit"?

[h0jeZvgoxFepBQ2C]

I really don't understand why docker is not looking into this, I guess there are so many companies out there which are pushing thousands of GB all the time just because of this bug. I guess they could save really a huge amount of bandwidth / traffic.

[h0jeZvgoxFepBQ2C]

@thaJeztah are you aware that this bug probably causes Petabyte of unnecessary traffic for your company?

[h0jeZvgoxFepBQ2C]

@thaJeztah i really can't emphasize the importance of this issue. I really can't understand why this issue gets ignored for such a long time. Is there anyone from the docker or moby team here which took a look?

Docker is losing soo much money due to traffic due to this issue, it's really suprising for me. Would be great to get at least a statement? Also why the overlay1 driver was removed, since this has not been fixed at all?

[thaJeztah]

This may actually be resolved with the containerd image-store integration (https://docs.docker.com/storage/containerd/) enabled; also see this other ticket;

• Proposal: Allow "COPY --from" to skip files from base image #35280

Without the containerd image-store integration enabled, the example from #21950 (comment) above produces;

docker history test
IMAGE          CREATED         CREATED BY                          SIZE      COMMENT
8842f6ab9dda   1 second ago    COPY . / # buildkit                 100MB     buildkit.dockerfile.v0
<missing>      2 seconds ago   COPY . / # buildkit                 100MB     buildkit.dockerfile.v0
<missing>      9 months ago    BusyBox 1.36.1 (glibc), Debian 12   4.26MB

docker image ls
REPOSITORY   TAG       IMAGE ID       CREATED              SIZE
test         latest    8842f6ab9dda   About a minute ago   204MB
<none>       <none>    6af04fbd25b7   About a minute ago   104MB

With the containerd image-store integration enabled, the second copy only adds 10MB, not the full 100MB;

docker history test
IMAGE          CREATED              CREATED BY                          SIZE      COMMENT
00842ea4df96   About a minute ago   COPY . / # buildkit                 10MB      buildkit.dockerfile.v0
<missing>      About a minute ago   COPY . / # buildkit                 100MB     buildkit.dockerfile.v0
<missing>      9 months ago         BusyBox 1.36.1 (glibc), Debian 12   4.31MB

docker image ls
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
test         latest    00842ea4df96   2 minutes ago   117MB

To verify, do you have an example reproducer? Also if you have an environment to test in, it might be worth trying if the containerd image-store integration resolves your issue (https://docs.docker.com/storage/containerd/). It's possible that additional changes are still needed in BuildKit (https://github.com/moby/buildkit) which is used as default builder for building images.

[h0jeZvgoxFepBQ2C]

Can I try this by enabling this checkbox in latest stable Docker Desktop on OSX? Or is this something different?

[thaJeztah]

@h0jeZvgoxFepBQ2C that's the checkbox that enables it, yes. But take note of the description under it; the containerd image store uses a store that's separate from the "graph driver" store; it won't delete those existing images and containers, but they will be "invisible" (and still take up space in your docker desktop storage).

So if you don't have "pet" containers and images, you could choose to first remove your containers and images, or do a "purge data" or "factory reset" (from the "troubleshoot" page in Docker Desktop's UI) ⚠️ this will remove your content (containers, images, volumes) ⚠️.

[h0jeZvgoxFepBQ2C]

Sorry for the long delay in my reply.. I try to enable the checkbox, but then docker stops completely. In the logs I can only see:

time="2024-05-02T09:12:24.036832000Z" level=info msg="Loading containers: done."
time="2024-05-02T09:12:24.040745625Z" level=warning msg="WARNING: daemon is not using the default seccomp profile"
time="2024-05-02T09:12:24.040773250Z" level=info msg="Docker daemon" commit=8b79278 containerd-snapshotter=true storage-driver=overlayfs version=26.0.0
time="2024-05-02T09:12:24.041105292Z" level=info msg="Daemon has completed initialization"
invalid UUID length: 0: unknown