File disappears after extracting with ADD

[jordi-chacon]

Hi,

jordi@burru:~$ uname -a
Linux burru 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

jordi@burru:~$ docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

jordi@burru:~$ docker -D info
Containers: 89
Images: 298
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 476
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.16.0-30-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 7.506 GiB
Name: burru
ID: ECBY:TFZS:OJF2:NKF5:SPAC:SCHI:YQ7Z:JV5O:2K7Z:MCAB:Z5ZZ:TJSC
Debug mode (server): false
Debug mode (client): true
Fds: 20
Goroutines: 31
System Time: Mon Aug 24 20:06:41 CEST 2015
EventsListeners: 0
Init SHA1: 7f9c6798b022e64f04d2aff8c75cbf38a2779493
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
Username: jordichacon
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support

I have a tar.gz file containing a bunch of Erlang binaries that make up my application binary. I also have a Dockerfile which, among other things, does ADD binary.tar.gz /service/. This is supposed to extract the contents of the tar.gz inside the /service directory. After extraction, 2 out 1269 files are missing.

Just for double checking, I have edited my Dockerfile to do the following

ADD binary.tar.gz /service1/
COPY binary.tar.gz /service2/

After building the image, I launch a container running it, and manually decompress the tar.gz in /service2 with tar -xf binary.tar.gz.

Then I run diff --brief -r /service1 /service2 which tells me that /service2 contains two files that do not exist in /service1. These files are regular files, not symlinks.

I would love to send you the tar.gz for you to test it yourself but it is not public.

Jordi

[GordonTheTurtle]

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

---

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

[cpuguy83]

Can you provide information about these two files? How the tar was created? etc?

[jordi-chacon]

Thank you for the quick reply!

I have managed to create a public dummy Elixir application which reproduces the problem. You can find it here: https://github.com/jordi-chacon/dummy_phoenix_app

You will need to install Elixir (and Erlang) to create the binary. In case you don't want to do that, I have also added a binary (binary.tar.gz) generated by my machine as part of the repo. You can use it to reproduce the problem.

To reproduce the problem, simply run make test_missing_files.

The tar is created by exrm, a release manager library for Elixir which I merely use to pack my application. I will let the owner of that library know about this issue, hopefully he will add his input here.

Thank you for your help.

[bitwalker]

@cpuguy83 I can provide more information on the tar file. It is created using Erlang's standard library erl_tar module, with the compressed option. Basically this means that erl_tar will create a tarball with the provided files, and then gzip it. That's about it.

erl_tar has been around a long time and is well tested, and the tarballs produced by exrm (the library I wrote which @jordi-chacon is using to produce his release) has so far worked fine with tar -xf on all the platforms I've seen in use with Elixir releases (Debian, Ubuntu, CentOS, OSX, Windows, RaspPi). If there is any additional information I can provide, I'll keep an eye on this thread, so just let me know!

[cbrunnkvist]

Due to change in behavior in Docker 0.x-something anno 2014, the ADD command is still causing confusion.

I think that it is the documentation that is wrong, since it only makes sense for the command to do the same post-copy processing on both local and remote files. This paragraph in https://docs.docker.com/engine/reference/builder/#add:

If <src> is a local tar archive in a recognized compression format (identity, gzip, bzip2 or xz) then it is unpacked as a directory. Resources from remote URLs are not decompressed.

should be corrected to read:

If <src> points to a local or remote tar archive in a recognized compression format (identity, gzip, bzip2 or xz) then it is unpacked as a directory.

That would enable us to close roughly 20 tickets spanning docker and moby projects all concerning the same issue ;)

[thaJeztah]

@cbrunnkvist extraction of remote archives is a bug in 17.06.0, and will be fixed in 17.06.1; see the release notes; https://github.com/docker/docker-ce/releases/tag/v17.06.0-ce

NOTE: Docker 17.06.0 has an issue in the image builder causing a change in the behavior of the ADD instruction of Dockerfile when referencing a remote .tar.gz file. The issue will be fixed in Docker 17.06.1.

see #33849 and #33851 for the fix

The problem discussed in this issue could possibly be related to a bug in Go that got fixed recently

[thaJeztah]

closing as this one went stale