Behaviour of ADD for remote tar balls changed with 17.06

[rn]

It looks like the behaviour of ADD with remote URLs changed with 17.06 onwards (also in DfM 17.07.0-rc1-ce-mac18) and is inconsistent with the documentation.

Here is a small example:

FROM alpine:3.6
ENV pkgname=rng-tools pkgver=5
ADD http://downloads.sourceforge.net/project/gkernel/$pkgname/$pkgver/$pkgname-$pkgver.tar.gz $pkgname-$pkgver.tar.gz

using:

docker build -t foo -f Dockerfile .
docker run --rm -ti foo ls -l rng-tools-5.tar.gz

With version earlier than 17.06 you see the documented behaviour:

$ docker run --rm -ti foo ls -l rng-tools-5.tar.gz
-rw-------    1 root     root        122182 Apr 18  2014 rng-tools-5.tar.gz

ie the remote URL is simply downloaded and not upacked.

Starting with 17.06 (I'm using 17.06.0-ce-rc5-mac16 or the latest edge release), I get:

$ docker run --rm -ti foo ls -l rng-tools-5.tar.gz
total 4
drwxr-xr-x    3 1000     1000          4096 Apr 18  2014 rng-tools-5

So here the tar ball has been unpacked into a directory.

rn-mbp:/Users/rneugeba/tmp: docker version
Client:
 Version:      17.06.0-ce-rc5
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   b7e4173
 Built:        Tue Jun 20 07:14:13 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.0-ce-rc5
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   b7e4173
 Built:        Tue Jun 20 07:18:21 2017
 OS/Arch:      linux/amd64
 Experimental: true
rn-mbp:/Users/rneugeba/tmp: docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 17.06.0-ce-rc5
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.31-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952GiB
Name: moby
ID: MWOK:F4YQ:OPXT:HV6H:4YTS:4YEB:6JOK:PYDN:HXLI:4667:2YMS:OKDL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 22
 Goroutines: 30
 System Time: 2017-06-27T16:13:51.731932221Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[justincormack]

cc @tonistiigi

[ph-One]

Ah! I noticed and utilized this behavior -- it was a welcomed addition for me 😞 . Saved me from having to do the whole wget && tar && rm bit. This means that I have to apk add openssl to my images to wget over https, or apk add curl, OR use multistage where my first build contains the messy wget and the second is ADD --from... either way, my short week just got a bit longer.

[thaJeztah]

@ph-One sorry for that 😅

[tomc974]

Just noticed this behavior today and like @ph-One, I really like it, even though it broke all my Dockerfiles. It allows me to reduce the size of my images by 4 MB.

You should really consider to keep this feature but in a backward compatible way. What about ADD --unpack <url> <dest>?

[chrisdlangton]

looks to be broken again in 17.09.0-ce

FROM alpine:latest
WORKDIR /tmp
ADD https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz ./
RUN ls -la .

outputs

~$ docker build -t add-test .
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM alpine:latest
 ---> 053cde6e8953
Step 2/4 : WORKDIR /tmp
 ---> 05b6ee81f5e3
Removing intermediate container 120e97373624
Step 3/4 : ADD https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz ./
Downloading [==================================================>]   89.8MB/89.8MB

 ---> 7719aea6804e
Step 4/4 : RUN ls -la .
 ---> Running in c19d166a59f7
total 87708
drwxrwxrwt    1 root     root          4096 Nov 14 04:00 .
drwxr-xr-x    1 root     root          4096 Nov 14 04:00 ..
-rw-------    1 root     root      89803580 Feb 16  2017 go1.8.linux-amd64.tar.gz
 ---> 0bf1301dcd7b
Removing intermediate container c19d166a59f7
~$ docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:09 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:45:38 2017
 OS/Arch:      linux/amd64
 Experimental: false

[thaJeztah]

@chrisdlangton you're describing the correct behavior; remote tar balls should not be extracted (which is what your example is showing)

[jmon12]

I actually find the "wget && tar" (ex-)feature of ADD quite useful. For what reason is it not kept, or available as an option (like @tomc974 mentioned)?

I just read in the doc with disappointment that it should actually not be extracted. My usecase is the following: a very big dir needs to be added to the image and I'd like the user building the image to be able to choose between downloading the big tar or using his local tar within the context... (inconvenience of the URL if low bandwidth etc...)
It would be very elegant and convenient with ADD's "wget && tar" feature which would induce a switch with an ARG instruction as follows, where the default value would be either an URL or a local path:

ARG flag=default_value
ADD ${flag} /path/to/dest