upload_project fails

[aidanmelen]

I am simply trying to copy a directory from my local machine to a virtual machine using fabric.contrib.project.upload_project feature.

my environment is:
python==2.7.12
fabric==1.13.1

here is the directory structure :
.
├── .vagrant
│   └── ...
├── Vagrantfile
├── .local_dir
│   └── file.txt
└── upload_project_test.py

upload_project_test.py contains the following code

from fabric.contrib.project import upload_project
from fabric.state import env
from os import path, getcwd, sep
from fabric.context_managers import lcd
from fabric.operations import local

project_root_path = getcwd()
ssh_config = None

with lcd(path.join(project_root_path)):
    if 'running' in local('vagrant status', capture=True):
        ssh_config_lines = local('vagrant ssh-config', capture=True)
        ssh_config = dict(line.split() for line in ssh_config_lines.split('\n'))

env.user = ssh_config['User']
env.key_filename = ssh_config['IdentityFile']
env.host_string = '{user}@{host}:{port}'.format(
                                   user=ssh_config['User'],
                                   host=ssh_config['HostName'],
                                   port=ssh_config['Port'])

local_dir = path.join(project_root_path, '.local_dir')
remote_dir = path.join(sep)    # vm root

upload_project(local_dir=local_dir, remote_dir=remote_dir, use_sudo=True)

this will fail because .local dir is hidden.

[localhost] local: vagrant status
[localhost] local: vagrant ssh-config
[localhost] local: tar -czf /var/folders/8z/dm68k1vd1zxf73slr540s3wc0000gn/T/tmpMGJuIH/local_dir.tar.gz -C /Users/aidanmelen/Sampleminded/infrastructure/python_tools/tastetest/test local_dir
tar: local_dir: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.

Fatal error: local() encountered an error (return code 1) while executing 'tar -czf /var/folders/8z/dm68k1vd1zxf73slr540s3wc0000gn/T/tmpMGJuIH/local_dir.tar.gz -C /Users/aidanmelen/Sampleminded/infrastructure/python_tools/tastetest/test local_dir'

Aborting.
[localhost] local: rm -rf /var/folders/8z/dm68k1vd1zxf73slr540s3wc0000gn/T/tmpMGJuIH

it will work properly if '.local_dir; is not hidden, such as 'local_dir'

[ploxiln]

It looks like you specify "local_dir" instead of the actual name of the folder ".local_dir" e.g.

local_dir = path.join(project_root_path, 'local_dir')

should be

local_dir = path.join(project_root_path, '.local_dir')

[aidanmelen]

(update) Well yes, that's true, Using 'local_dir' will pass. The problem is that upload_project cannot handle hidden directories. For this reason, '.local_dir would fail.

[ploxiln]

Seems to handle hidden directories for me:

import os

from fabric.api import task, run
from fabric.contrib.project import upload_project

@task
def test():
    local_dir = os.path.abspath(".local_dir")
    upload_project(local_dir=local_dir, remote_dir="/tmp", use_sudo=True)

$ fab --version
Fabric 1.13.1
Paramiko 2.1.1
$ ls -la .local_dir
total 16
drwxr-xr-x   4 pierce  staff  136 Mar  4 12:38 .
drwxr-xr-x  12 pierce  staff  408 Mar  4 12:44 ..
-rw-r--r--   1 pierce  staff   52 Mar  4 12:38 testfile.log
-rw-r--r--   1 pierce  staff   11 Mar  4 12:38 testfile.txt
$ fab --show=debug -H testdeploy03.ec2.st-av.net test
Using fabfile '/Users/pierce/scratch/fabwork/fabfile.py'
Commands to run: test
Parallel tasks now using pool size of 1
[testdeploy03.ec2.st-av.net] Executing task 'test'
[localhost] local: tar -czf /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmpOMtLKB/.local_dir.tar.gz -C /Users/pierce/scratch/fabwork .local_dir
[testdeploy03.ec2.st-av.net] put: /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmpOMtLKB/.local_dir.tar.gz -> /tmp/.local_dir.tar.gz
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "mv \"7b854f99c175b8342707837da591666e8dcea132\" \"/tmp/.local_dir.tar.gz\""
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "cd /tmp >/dev/null && tar -xzf .local_dir.tar.gz"
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "cd /tmp >/dev/null && rm -f .local_dir.tar.gz"
[localhost] local: rm -rf /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmpOMtLKB

Done.
Disconnecting from testdeploy03.ec2.st-av.net... done.

upload_project() does have the strange problem with needing local_dir to contain a path separator in it (so making it an absolute path works, which is equivalent to prepending with getcwd() as you did). But this function, along with most of "contrib", will most likely be dropped in fabric 2.0.

[aidanmelen]

Gotcha @ploxiln, I didn't realize that upload_project() required absolute paths for arguments. I will give that a try.

[ploxiln]

In your original report, you did specify an absolute path, but you did not specify the correct folder name.

In fact, an absolute path is not needed, just any path with more than a single component (meaning, with a slash). This works:

from fabric.api import env, task, run, puts
from fabric.contrib.project import upload_project

@task
def test():
    upload_project("./.local_dir", "/tmp", use_sudo=True)

$ fab --show=debug -H testdeploy03.ec2.st-av.net test
Using fabfile '/Users/pierce/scratch/fabwork/fabfile.py'
Commands to run: test
Parallel tasks now using pool size of 1
[testdeploy03.ec2.st-av.net] Executing task 'test'
[localhost] local: tar -czf /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmp1mfo7q/.local_dir.tar.gz -C . .local_dir
[testdeploy03.ec2.st-av.net] put: /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmp1mfo7q/.local_dir.tar.gz -> /tmp/.local_dir.tar.gz
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "mv \"7b854f99c175b8342707837da591666e8dcea132\" \"/tmp/.local_dir.tar.gz\""
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "cd /tmp >/dev/null && tar -xzf .local_dir.tar.gz"
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "cd /tmp >/dev/null && rm -f .local_dir.tar.gz"
[localhost] local: rm -rf /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmp1mfo7q

[aidanmelen]

okay thanks for your input. I think this issue can be closed.