New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File provisioner doesn't upload directories when using Docker #5390

Closed
hubertgrzeskowiak opened this Issue Sep 27, 2017 · 3 comments

Comments

Projects
None yet
2 participants
@hubertgrzeskowiak
Copy link

hubertgrzeskowiak commented Sep 27, 2017

When using Docker, it's impossible to upload whole directories. The file provisioner will only upload the contents of a given directory.

  • Packer version from packer version
    Packer v1.1.0
  • Host platform
    Mac OS Sierra 10.12.6
  • Debug log output from PACKER_LOG=1 packer build template.json.
    Please paste this in a gist https://gist.github.com
    let me know if this is really necessary
  • The simplest example template and scripts needed to reproduce the bug.
{
  "_comment": "Docker test",
  "builders": [
    {
      "type": "docker",
      "image": "ubuntu",
      "commit": true
    }
  ],
  "provisioners": [
    {
      "type": "file",
      "source": "{{template_dir}}/image",
      "destination": "/home/ubuntu/"
    },
    {
      "type": "shell",
      "inline": [
        "ls /home/ubuntu"
      ]
    },

According to docs, if source directory is not written with a slash at the end, the whole directory should be moved to the destination. This does work with AWS builder (I tested), but with Docker it only moves the contents of image in this case. The ls above prints the contents of image but not image itself. Appending a slash on the source makes no difference.

Here's the corresponding output lines for with and without the slash:
==> docker: Uploading /Users/hubertgrzeskowiak/infrastructure/app/cs/ami/image => /home/ubuntu/
==> docker: Uploading /Users/hubertgrzeskowiak/infrastructure/app/cs/ami/image/ => /home/ubuntu/

The output of ls is the same in both cases.

I have also tried using /home/ubuntu/image as the destination, but this causes the build to fail:

==> docker: Uploading /Users/hubertgrzeskowiak/infrastructure/app/cs/ami/image => /home/ubuntu/image
==> docker: Killing the container: ae74c48e633cb15991ac19e7a4592736a8bf7bba9b029d34d781fb7d4547270a
Build 'docker' errored: Failed to upload to '/home/ubuntu/image' in container: Error response from daemon: lstat /var/lib/docker/overlay2/80bf06ce5a8eb7b04b517905ae0a487da90be4b5a96273faacc7dfdd21957d6a/merged/home/ubuntu: no such file or directory
. exit status 1.

The documentation says one should not rely on the file provisioner creating directories in Docker, but it also says that it should work.

@hubertgrzeskowiak hubertgrzeskowiak changed the title The file provisioner cannot upload directories when using Docker File provisioner doesn't upload directories when using Docker Sep 27, 2017

@mwhooker

This comment has been minimized.

Copy link
Collaborator

mwhooker commented Sep 27, 2017

I'm afraid I'm unable to reproduce this on osx with ubuntu. Can you make sure you're on the latest version of docker?

==> docker: Creating a temporary directory for sharing data...
==> docker: Pulling Docker image: ubuntu:xenial
    docker: xenial: Pulling from library/ubuntu
    docker: 9fb6c798fa41: Pulling fs layer
    docker: 3b61febd4aef: Pulling fs layer
    docker: 9d99b9777eb0: Pulling fs layer
    docker: d010c8cf75d7: Pulling fs layer
    docker: 7fac07fb303e: Pulling fs layer
    docker: 7fac07fb303e: Waiting
    docker: d010c8cf75d7: Waiting
    docker: 9d99b9777eb0: Verifying Checksum
    docker: 3b61febd4aef: Verifying Checksum
    docker: 9d99b9777eb0: Download complete
    docker: 3b61febd4aef: Download complete
    docker: d010c8cf75d7: Download complete
    docker: 7fac07fb303e: Download complete
    docker: 9fb6c798fa41: Verifying Checksum
    docker: 9fb6c798fa41: Download complete
    docker: 9fb6c798fa41: Pull complete
    docker: 3b61febd4aef: Pull complete
    docker: 9d99b9777eb0: Pull complete
    docker: d010c8cf75d7: Pull complete
    docker: 7fac07fb303e: Pull complete
    docker: Digest: sha256:d45655633486615d164808b724b29406cb88e23d9c40ac3aaaa2d69e79e3bd5d
    docker: Status: Downloaded newer image for ubuntu:xenial
==> docker: Starting docker container...
    docker: Run command: docker run -v /Users/mwhooker/.packer.d/tmp/packer-docker811785259:/packer-files -d -i -t ubuntu:xenial /bin/bash
    docker: Container ID: f59b05f99fe3fa77dba605f4d5f495180b2a59236516fe21a6fc0d9105be9b3d
==> docker: Uploading ./FILES => /tmp/
==> docker: Provisioning with shell script: /var/folders/yy/ff0tkhr141xdxvxf5fz43kmw0000gn/T/packer-shell785717260
    docker: total 8
    docker: drwxr-xr-x 2 501 dialout 4096 Aug 24 00:28 FILES
    docker: -rwxr-xr-x 1 501 dialout   24 Sep 27 16:39 script_2232.sh
==> docker: Committing the container
    docker: Image ID: sha256:1308b4d791c6c6e8ebdcd1ac229f3145001920610e6a81f7f49c6c3b0778cec8
==> docker: Killing the container: f59b05f99fe3fa77dba605f4d5f495180b2a59236516fe21a6fc0d9105be9b3d
Build 'docker' finished.

==> Builds finished. The artifacts of successful builds are:
--> docker: Imported Docker image: sha256:1308b4d791c6c6e8ebdcd1ac229f3145001920610e6a81f7f49c6c3b0778cec8
cat dockercp.json

{
    "builders": [
        {
            "commit": true,
            "image": "ubuntu:xenial",
            "type": "docker"
        }
    ],
    "provisioners": [
        {
            "type": "file",
            "source": "./FILES",
            "destination": "/tmp/"
        },
        {
            "type": "shell",
            "inline": ["ls -l /tmp"]
        }
    ]
}

@mwhooker mwhooker added the need-repro label Sep 27, 2017

@hubertgrzeskowiak

This comment has been minimized.

Copy link

hubertgrzeskowiak commented Sep 28, 2017

I think I found the reason for this. The ubuntu docker image does not contain a /home/ubuntu directory. Therefore the file provisioner assumed I wanted to rename my directory into ubuntu. When creating the directory beforehand, the behavior was correct.

First, the destination directory must already exist. If you need to create it, use a shell provisioner just prior to the file provisioner in order to create the directory. If the destination directory does not exist, the file provisioner may succeed, but it will have undefined results. Note that the docker builder does not have this requirement. It will create any needed destination directories, but it's generally best practice to not rely on this behavior.

The last two sentences of the quoted docs above can be removed I think, as they are misleading.

Thanks and cheers

@mwhooker

This comment has been minimized.

Copy link
Collaborator

mwhooker commented Oct 3, 2017

good idea. pushed in e26df7e

@mwhooker mwhooker closed this Oct 3, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment