"COPY foo/* bar/" not work if no file in foo/

[icy]

Example:

Step 0 : COPY source/* /source/
INFO[0001] No source files were specified

If there isn't any file in source/ directory, the build will fail.

Expected behavior: Continue if no file matches source/*.

This is very annoyed when COPY is provided as ON BUILD instruction.

Docker information: https://github.com/icy/docker/blob/master/bugs/volume_bug/docker.info.txt

[icy]

OK here it is

$ docker version

Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64

$ docker info

Containers: 4
Images: 286
Storage Driver: devicemapper
 Pool Name: docker-8:4-9437191-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 4.264 GB
 Data Space Total: 107.4 GB
 Data Space Available: 103.1 GB
 Metadata Space Used: 10.48 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.137 GB
 Udev Sync Supported: true
 Data loop file: /home/locker/data/devicemapper/devicemapper/data
 Metadata loop file: /home/locker/data/devicemapper/devicemapper/metadata
 Library Version: 1.02.93 (2015-01-30)
Execution Driver: native-0.2
Kernel Version: 3.18.6-1-ARCH
Operating System: Arch Linux
CPUs: 4
Total Memory: 3.563 GiB
Name: icy
ID: TEOO:JOAR:QIJU:SZUF:YB4J:OUSB:YZM6:LYFA:FP5V:NVGL:4YZO:CDDB

$ uname -a

Linux icy 3.18.6-1-ARCH #1 SMP PREEMPT Sat Feb 7 08:44:05 CET 2015 x86_64 GNU/Linux

[icy]

If you are reporting a new issue, PLEASE make sure that does not have any duplicates already open.

How? scanning all > 800 tickets? Where is QA/ tester team of Docker?

[duglin]

#dibs

[duglin]

I believe this was intentional. Try:

$ cp foo* /tmp/
cp: cannot stat ‘foo*’: No such file or directory
$ echo $?
1

Doing what we see on the cmd line makes sense.

That being said, I can understand that this might cause some issues for people since, unlike in a script, you don't have the option of telling Docker's build process to continue.

I think the options are:
1 - do nothing, leave it as is
2 - don't make this a build-stopping error (perhaps keep the info msg for debugging purposes)
3 - add a --ignore-missing type of option to COPY/ADD so the user can explicitly tell us they're ok with missing source files. E.g. COPY --ignore-missing source/* source/

I'm ok with either 2 or 3, with a slight preference for 3 because doing 2 would be changing the current semantics and might break people.

ping @tiborvass @erikh @tianon

[tiborvass]

I tried cp foo/* bar/ where both foo and bar are empty directories and it fails.

I assume you are doing COPY source/* /destination/ because /destination/ already has files and you want to merge the two directories. If that's not the case and /destination/ is empty, simply do COPY source /destination.

@tianon do you think we should change this to a noop when source/ is empty ? If not, what would be an alternative solution for the OP ? I can think of terrible ON BUILD chains but I don't dare writing them.

[icy]

I tried cp foo/* bar/ where both foo and bar are empty directories and it fails.

There are many ways to ignore that error in system shell. There are also checks before cp. What I expected here is something like

COPY foo/* /bar/ || CONTINUE

# Maybe || is too Bashy, so we may take the idea from PHP
COPY foo/* /bar/ ||| CONTINUE

# Or even with warning
COPY foo/* /bar/ ||| CONTINUE with warning

I assume you are doing COPY source/* /destination/ because /destination/ already has files and you want to merge the two directories

Almost true. I have a base image which has a volume for future files. This volume actually has some initial files. The descendant may add some new things to that volume, or add nothing (Think, there are many applications can use this idea: nginx, apache, supervisor, mysql, php, bla bla.

What I have to do now is to create an empty file

# In the base image
ON BUILD copy foo/* bar/

# In the descendant image
# mkdir foo/ && touch foo/empty
# other stuff

[duglin]

If we want a way to ignore errors, then I'd prefer a more generic solution that doesn't invent new sh-like stuff. For example, for a while now I thought that having RUN --ignore-error some-bad-cmd... would be useful but didn't have a concrete need for it yet, so I didn't say anything. But there's no reason why --ignore-error couldn't be available on some Dockerfile commands, like COPY, if there was really a need for it.

[icy]

Yeah I need that. I don't know how to make it "really need" though ;)

[icy]

No worry. I can solve the problem with Bocker. I can even make my Dockerfiles modulable. See for example https://github.com/icy/bocker/blob/master/examples/Bockerfile.minimal.

Hope that's generic enough.

You may close this issue.

Thank you.

[mattmb]

We were also thinking about this the other day. What I'd like to do is provide a hook into an upstream container using ONBUILD. So something like:

ONBUILD COPY pre-install.sh
ONBUILD RUN /bin/bash -c "if [ -x pre-install.sh ];\
                          then \
                              pre-install.sh;\
                          fi"

However currently the COPY will always fail if the file doesn't exist (even if it's ONBUILD COPY *.sh)

We thought about templating too but in this example each application is inheriting this container so it would be more difficult.

I like the idea of a flag to ignore COPY errors.

[duglin]

@mattmb that's an interesting use-case.

[icy]

Thank you @duglin and the team for your work!

[bfirsh]

#13813 has been closed because the builder is in the process of being separated out, but this is still an issue. Reopening so we can track and incorporate into the new builder when that exists.

/cc @ashb @mattmb

[mattrobenolt]

+1 for being able to support something like this. This is super valuable for ONBUILD COPY as pointed out above.

So I'm happy to see this is being addressed. :)

[mattmb]

Thanks @bfirsh appreciate keeping a track on this one 👍

[duglin]

as much as I agree with keeping issues open, even if we can't work on them yet, so we don't lose track of them, I believe the current mode of operation is to close things we can't take action on. Which is why a lot of other Dockerfile/build issues have been closed. So I think we (sadly) need to close this for now.

pinging @icecrime for confirmation though.

[gioppoluca]

This happens on many official images like fluend one

[shabbychef]

+1 for the ONBUILD COPY use case...

[GordonTheTurtle]

USER POLL

The best way to get notified of updates is to use the Subscribe button on this page.

Please don't use "+1" or "I have this too" comments on issues. We automatically
collect those comments to keep the thread short.

The people listed below have upvoted this issue by leaving a +1 comment:

@loretoparisi

[duglin]

What do people think about reopening #13813 (adding a --ignore-error type of flag) since the builder restrictions are now gone?

ping @docker/maintainers

[moshebeeri]

it is just a bug!!!

ubuntu@nodejs:~/test$ docker build --no-cache -t lowla/trender .
Sending build context to Docker daemon  2.56 kB
Step 1 : FROM ubuntu
 ---> 0ef2e08ed3fa
Step 2 : COPY ../id_rsa* /
No source files were specified
ubuntu@nodejs:~/test$ cp ../id_rsa* .
ubuntu@nodejs:~/test$ nano Dockerfile 
ubuntu@nodejs:~/test$ docker build --no-cache -t lowla/trender .
Sending build context to Docker daemon 3.584 kB
Step 1 : FROM ubuntu
 ---> 0ef2e08ed3fa
Step 2 : COPY id_rsa* /
 ---> 98b111084af3
Removing intermediate container 39ba1bcc02cf
Successfully built 98b111084af3
ubuntu@nodejs:~/test$ 

Why can't I copy from parent directory?

[ashb]

@moshebeeri This seems to be a different bug, but you can't copy files from outside the build context by design so this shouldn't work: Form Dockerfile reference:

COPY obeys the following rules:

• The <src> path must be inside the context of the build; you cannot COPY ../something /something, because the first step of a docker build is to send the context directory (and subdirectories) to the docker daemon.

I guess I would have expected it to complain about the src being out of the build context though.

[cpuguy83]

@moshebeeri Because it's not part of the build context. This is intentional behavior, also doesn't really seem related to this issue.

side note, what you are doing in that test is a really bad idea. You never want to copy secrets into an image like that as there is no way to remove them from the resulting image. Even if you rm my_secret it's still going to be part of the image itself, just not in the resulting container rootfs.

[dashesy]

A Try/Catch block would solve this and will be useful for other purposes too:

TRY
COPY foo/* /bar/
CATCH
COPY baz/* /bar/

[moshebeeri]

I disagree, in the case presented I wanted to copy a private key.
It is worse idea to keep keys in the code, Though there are work around it should not be a restriction by docker in the first place.
Thanks for the help @cpuguy83 @ashb and @dashesy

[thaJeztah]

Discussing this in the maintainers session, and we think having the COPY --mount option would solve this use case (and many others); #32507. Contributions for that are welcome (but @tonistiigi mentions that once we switch to the new BuildKit, implementing would be quite easy 😇 )

[thaJeztah]

I'm keeping this issue open, but this will be resolve if the other proposal is implemented 👍

[tomaszalusky]

COPY source/. /source/ works for me (i.e. copies directory when empty or not). Details here.

[felipecrs]

Here is my suggestion:

# syntax=docker/dockerfile:1.2

RUN --mount=type=bind,source=jars,target=/build/jars \
  find /build/jars -type f -name '*.jar' -maxdepth 1  -print0 \
  | xargs -0 --no-run-if-empty --replace=source cp --force source "${INSTALL_PATH}/modules/"

That works around:

COPY jars/*.jar "${INSTALL_PATH}/modules/"

But copies no *.jar if none is found, without throwing an error.

[maffe]

I assume you are doing COPY source/* /destination/ because /destination/ already has files and you want to merge the two directories. If that's not the case and /destination/ is empty, simply do COPY source /destination.

In my case the use of the wildcard is required because I don’t want to copy all files:

$ cat Dockerfile
FROM alpine:3
COPY import/*.sql /tmp/import/
ENTRYPOINT [ "ls", "/tmp/import/" ]

$ ls import/
maybe-present.sql  Readme.txt

$ docker build -t test . && docker run --rm test
[+] Building 7.5s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 121B                                                                               0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/library/alpine:3                                                        7.5s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 69B                                                                                   0.0s
 => [1/2] FROM docker.io/library/alpine:3@sha256:02bb6f428431fbc2809c5d1b41eab5a68350194fb508869a33cb1af4444c9b11  0.0s
 => => resolve docker.io/library/alpine:3@sha256:02bb6f428431fbc2809c5d1b41eab5a68350194fb508869a33cb1af4444c9b11  0.0s
 => CACHED [2/2] COPY import/*.sql /tmp/import/                                                                    0.0s
 => exporting to image                                                                                             0.0s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:8f52835b04b8e0cd478156a3684d2af7f2926161dac3d873a377acfbee1f2941                       0.0s
 => => naming to docker.io/library/test                                                                            0.0s
maybe-present.sql

$ rm import/maybe-present.sql

$ ls import/
Readme.txt

$ docker build -t test . && docker run --rm test
[+] Building 0.8s (6/6) FINISHED
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 121B                                                                               0.0s
 => [internal] load metadata for docker.io/library/alpine:3                                                        0.7s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => CACHED [1/2] FROM docker.io/library/alpine:3@sha256:02bb6f428431fbc2809c5d1b41eab5a68350194fb508869a33cb1af44  0.0s
 => => resolve docker.io/library/alpine:3@sha256:02bb6f428431fbc2809c5d1b41eab5a68350194fb508869a33cb1af4444c9b11  0.0s
 => ERROR [2/2] COPY import/*.sql /tmp/import/                                                                     0.0s
------
 > [2/2] COPY import/*.sql /tmp/import/:
------
Dockerfile:2
--------------------
   1 |     FROM alpine:3
   2 | >>> COPY import/*.sql /tmp/import/
   3 |     ENTRYPOINT [ "ls", "/tmp/import/" ]
   4 |
--------------------
ERROR: failed to solve: lstat /var/lib/docker/tmp/buildkit-mount813963266/import: no such file or directory

My current workaround is to remove unwanted files in an additional stage:

FROM alpine:3 AS prepare-import
COPY import/ /tmp/import/
RUN find /tmp/import/ -not -name \*.sql -type f -delete

FROM alpine:3
COPY --from=prepare-import /tmp/import/ /tmp/import/
ENTRYPOINT [ "ls", "/tmp/import/" ]