COPY with excluded files is not possible

[bronger]

I need to COPY a part of a context directory to the container (the other part is subject to another COPY). Unfortunately, the current possibilities for this are suboptimal:

1. COPY and prune. I could remove the unwanted material after an unlimited COPY. The problem is that the unwanted material may have changed, so the cache is invalidated.
2. COPY every file in a COPY instruction of it own. This adds a lot of unnecessary layers to the image.
3. Writing a wrapper around the "docker build" call that prepares the context in some way so that the Dockerfile can comfortably copy the wanted material. Cumbersome and difficult to maintain.

[cpuguy83]

See https://docs.docker.com/reference/builder/#dockerignore-file
You can add entries to a .dockerignore file in the root of the project.

[bronger]

.dockerignore does not solve this issue. As I wrote, "the other part is subject to another COPY".

[cpuguy83]

So you want to conditionally copy based on some other copy?

[bronger]

The context contains a lot of directories A1...A10 and a directory B. A1...A10 have one destination, B has another:

COPY A1 /some/where/A1/
COPY A2 /some/where/A2/
...
COPY A10 /some/where/A10/
COPY B some/where/else/B/

And this is awkward.

[cpuguy83]

What part of it is awkward? Listing them all individually?

COPY A* /some/where/
COPY B /some/where/else/

Does this work?

[bronger]

The names A1..A10, B were fake. Besides, COPY A* ... throws together the contents of the directories.

There are a couple of options I admit, but I think that all of them are awkward. I mentioned three in my original posting. A fourth option is to rearrange my source code permanently so that A1..A10 are moved in a new directory A. I was hoping that this was not necessary because an additional nesting level is not something to wish for, and my current tools needed to special-case my dockerised projects then.

(BTW, #6094 (following symlinks) would help in this case. But apparently, this is no option either.)

[cpuguy83]

@bronger if COPY behaved exactly like cp, would that solve your use-case?

I'm not sure I 100% understand.
Maybe @duglin can have a look.

[duglin]

@bronger I think @cpuguy83 asked the right question, how would you solve this if you were using 'cp' ? I looked and didn't notice some kind of excludes option on 'cp' so I'm not sure how you would solve this outside of a 'docker build' either.

[bronger]

With cp behaviour, I could ameliorate the situation by saying

COPY ["A1", ... "A10", "/some/where/"]

It's still a mild maintenance problem because I would have to think of that line if I added an "A11" directory. But that would be acceptable.

Besides, cp does not need excludes, because copying everything and removing the unwanted parts has almost no performance impact beyond the copying itself. With docker's COPY, it means wrongly invalidated cache every time B is changed, and bigger images.

[duglin]

@bronger you can do:

COPY a b c d /some/where

just like you were suggesting.

As for doing a RUN rm ... after the COPY ..., yes you'll have on extra layer, but you still should be able to use the cache. If you see a cache miss due to it let me know, I don't think you should.

[bronger]

But

COPY a b c d /some/where/

copies the contents of the directories a b c d together, instead of creating the directories /some/where/{a,b,c,d}. It works like rsync with a slash appended to the src directory. Therefore, the four instructions

COPY a /some/where/a/
COPY b /some/where/b/
COPY c /some/where/c/
COPY d /some/where/d/

are needed.

As for the cache ... if I say

COPY . /some/where/
RUN rm -Rf /some/where/e

then the cache is not used if e changes, although e is not effectively included into the operation.

[duglin]

@bronger yep, sadly you're correct. I guess we could add a --exclude zzz type of flag, but per https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax it may not get a lot of traction right now.

[bronger]

Fair enough. Then I will use a COPY+rm for the time being and add a FixMe comment. Thank you for your time!