"ADD <url> /" in Dockerfile copy the file instead of decompressing it

[sdouche]

Embarrassing bug, you must add the tarball in the repository.

[eliasp]

Ran into the same issue.
A minimal working example:

Dockerfile:

FROM stackbrew/ubuntu:13.10
ADD https://github.com/dotcloud/docker/archive/master.tar.gz /root/
CMD ["/bin/bash"]

Result:

$ docker build -t docker-add-test .
Uploading context 10240 bytes
Step 1 : FROM stackbrew/ubuntu:13.10
 ---> 784f46cb06bb
Step 2 : ADD https://github.com/dotcloud/docker/archive/master.tar.gz /root/
 ---> 3070132f0df1
Step 3 : CMD ["/bin/bash"]
 ---> Running in 2c975a819895
 ---> b939dcbd9d95
Successfully built b939dcbd9d95

$ docker run -i -t docker-add-test ls -l /root/
total 2180
-rw-r--r-- 1 root root 2230124 Oct 27 15:35 master.tar.gz

[crosbymichael]

@sdouche @eliasp

Where does it say in the docs that ADD decompresses archives?

Edit: Where does it say in the docs that ADD decompresses archives for downloads?

[vreon]

@crosbymichael IMO the docs don't make it clear that

If <src> is a URL and <dest> does not end with a trailing slash, then a file is downloaded from the URL and copied to <dest>.

takes precedence over

If <src> is a tar archive in a recognized compression format (identity, gzip, bzip2 or xz), it is unpacked as a directory.

[eliasp]

So it comes down to this two options:

• Make it more clear in the docs, that extracting archives works only for local files (I could create a PR to change this)
• Implement support for also extracting remote files (needs to be done by someone else)

[sdouche]

Extracting remote files would be great, it's far better than add tarball in the repo.

[crosbymichael]

My vote would be to update the docs. I don't think the cmd ADD should, add, download, detect decompression, decompress, etc. It should ..... ADD.

ping @vieux @metalivedev @shykes So far we have two votes and they both cancel each other out ;)

[tianon]

If we update the docs to not mention decompression, then we ought to also update the ADD command code to not decompress local archives, which will break our base image debootstrap Dockerfiles. The turtles at the bottom are biting at our heels. :)

[crosbymichael]

@tianon I didn't say remove the existing feature but just not implement it for URLS. ;)

[tianon]

Ahh, so you just want to clarify the existing documentation. I'm definitely cool with that. :) 👍

[runvnc]

For some strange reason I keep having to change my Dockerfile related to this tar.gz download:

ADD https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.3.tar.gz / #RUN tar xf elasticsearch-0.90.3.tar.gz RUN mv /elasticsearch-0.90.3.tar.gz/elasticsearch-0.90.3 /elasticsearch-0.90.3

So I have been commenting/uncommenting the last two lines. Seems like this morning I had to change it to be one way, and then this afternoon a different way. I don't know if docker is behaving differently on two different machines, or at different times, or if somehow the elasticsearch people keep uploading an archive structured differently, or if I am crazy.

[danielkza]

I find this discrepancy in behavior between remote and local files quite annoying. I'm generating a Dockerfile from a configuration file, and I'd like for a resource to be specified as either an URL, or a local file that was manually downloaded. Right now it's quite difficult to do so, because there is no valid combination that works for both.

ADD will extract from local files, but not from remote ones, meaning I don't know whether I should or not do the extraction manually. COPY does not support remote files.

I'd be really nice if COPY were made to support remote files, or if disabling extraction on ADD would be possible.

[tpaz]

+1 to danielkza comment. inconsistent behavior between local tar.gz files and remote ones

[isanych]

It would be nice to have ability to untar url

[justin8]

Definitely. Storing tarballs in git is rather obscene.

[ericvh]

++

[arnaudlamy]

+1 !

[anpieber]

although this had already been closed I'm also in strong favor of also extracting remote tar files instead of simply adding them.

[geerk]

How can I just add archive to the container without decompressing it?

EDIT: used COPY command. What a strange semantics...

[thaJeztah]

@geerk if it's a local file, COPY myfile /path/in/container in your Dockerfile. To download, RUN curl ... or similar.

But, please, the GitHub issue-tracker is not a support forum. Use #docker IRC channel on freenode or docker-user for support questions.

[xgamedevelop]

+1

[doomsbuster]

I am seeing a different issue where the ADD is copying the *.tar.gz file but it does not extract it in the subsequent add command. If you look at the logs below, it shows that the current working directory is /opt and that there is a file named 13.3.0.tar.gz but it still throws error indicating no such file or directory.

Step 15 : ADD https://github.com/xxx/xxx/archive/13.3.0.tar.gz /opt/
Downloading [==================================================>] 17.25 MB/17.25 MB
 ---> Using cache
 ---> d72baa355f8b
Step 16 : WORKDIR /opt
 ---> Using cache
 ---> 0a6bf0c4a90e
Step 17 : RUN pwd
 ---> Running in 8fe020c4b562
/opt
 ---> b8281e02a0de
Removing intermediate container 8fe020c4b562
Step 18 : RUN ls -lrt
 ---> Running in bc66eca2975c
total 16848
-rw------- 1 root root 17247625 Jan  1  1970 13.3.0.tar.gz
drwxr-xr-x 2 root root     4096 Dec 29 22:31 xxx
 ---> 25c7126b35e3
Removing intermediate container bc66eca2975c
Step 19 : ADD 13.3.0.tar.gz /opt/xxx/
lstat 13.3.0.tar.gz: no such file or directory```

[thaJeztah]

@doomsbuster is that file present in the same directory as your dockerfile? ADD looks for the .tar.gz file in the build context that was sent to the daemon during build, it does not copy the file from the container's filesystem

[doomsbuster]

@thaJeztah I am downloading the file in the /opt using ADD command and then trying to extract it using a second ADD command in my Dockerfile:

ADD https://github.com/xxx/xxx/archive/13.3.0.tar.gz /opt
ADD 13.3.0.tar.gz /opt/tcexam

[thaJeztah]

@doomsbuster that's not what ADD does; ADD copies files into the image, either from the local build context, or from a remote URL.

In your example, you're trying to copy a .tar.gz file into the image twice; one time from a remote URL, and a second time, from the local build context (i.e., by default; the directory that the Dockerfile is in). The second ADD fails of you don't have that file present locally

[doomsbuster]

aah! I see. That clarifies it. While we talk i did something like this to get it to work:

WORKDIR /opt
RUN wget https://github.com/xxx/xxx/archive/13.3.0.tar.gz && tar -zxvf 13.3.0.tar.gz

[thaJeztah]

Yes; that's probably the best option; as can be seen in the discussion above, ADD has a bit too much "magic", and at one point was marked for deprecation (in favor of COPY). In the end it was decided to not deprecate ADD, but to not add any more magic to it.

[danielporto]

2018 and this problem still annoys me >.<
please create a DOWNLOAD and EXTRACT commands so we know what it actually does.

[HuKeping]

So why do you guys remove the download and extract feature in image building at all?

[thaJeztah]

@HuKeping what do you mean; you mean removing the feature from ADD?

[HuKeping]

@thaJeztah it seems quite a lot of people want that feature(when download from URL, also extract it automatically) of ADD.

Recently I came into the same issue, my Dockerfile is somewhat like

From scratch

ADD URL/sle12sp2_kiwi.tar.xz

...

For some security reason they would prefer to ADD from a remote address which can be accessed by public so that they could generate the image wherever they want.

AFAIK, one of the reason why you drop this feature is because there is no guarantee how huge the package size might be and it might cause the docker daemon got killed and then other running container down. I'll buy this but I still think if people want shoot themselves in the foot, just let them go, it should not blame on the feature.

[thaJeztah]

We can't change the default behavior as it would break many Dockerfiles, and there's definitely use cases for adding a remote tar without decompressing it.

Would a multi-stage build solve your issue? i.e., decompress in an intermediate step, and copy to the final stage?

[HuKeping]

Would a multi-stage build solve your issue? i.e., decompress in an intermediate step, and copy to the final stage?

it not work if build from scratch.

We can't change the default behavior as it would break many Dockerfiles, and there's definitely use cases for adding a remote tar without decompressing it.

Definitely agree, I don't think it's a good idea to change the default behavior, I was thinking add some option to ADD to handle it.

[thaJeztah]

Also, if your use case is to create an image from a rootfs; you can use docker import; https://docs.docker.com/engine/reference/commandline/import/

[HuKeping]

docker import is help, thanks @thaJeztah .

Just wonder if there is a way to reproduce the environment(image) in one process.

[thaJeztah]

Perhaps when using DOCKER_BUILDKIT=1 and the experimental dockerfile syntax (https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md);

update this feature is no longer experimental so no longer needed to use the docker/dockerfile:experimental syntax

Here's one based on my "silly experiments with RUN --mount";

# syntax=docker/dockerfile:1
FROM scratch
ARG DOCKER_VERSION=18.09.1
RUN --mount=from=busybox:latest,src=/bin/,dst=/bin/ \
 wget -O - https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_VERSION}.tgz | tar zxf -

docker build -t myimage .

docker run --rm myimage docker/docker --version
Docker version 18.09.1, build 4c52b90

[HuKeping]

Thanks @thaJeztah , it's helpful!