New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

upload_project fails #1574

Closed
aidanmelen opened this Issue Mar 4, 2017 · 6 comments

Comments

Projects
None yet
2 participants
@aidanmelen

aidanmelen commented Mar 4, 2017

I am simply trying to copy a directory from my local machine to a virtual machine using fabric.contrib.project.upload_project feature.

my environment is:
python==2.7.12
fabric==1.13.1

here is the directory structure :
.
├── .vagrant
│   └── ...
├── Vagrantfile
├── .local_dir
│   └── file.txt
└── upload_project_test.py

upload_project_test.py contains the following code

from fabric.contrib.project import upload_project
from fabric.state import env
from os import path, getcwd, sep
from fabric.context_managers import lcd
from fabric.operations import local

project_root_path = getcwd()
ssh_config = None

with lcd(path.join(project_root_path)):
    if 'running' in local('vagrant status', capture=True):
        ssh_config_lines = local('vagrant ssh-config', capture=True)
        ssh_config = dict(line.split() for line in ssh_config_lines.split('\n'))

env.user = ssh_config['User']
env.key_filename = ssh_config['IdentityFile']
env.host_string = '{user}@{host}:{port}'.format(
                                   user=ssh_config['User'],
                                   host=ssh_config['HostName'],
                                   port=ssh_config['Port'])

local_dir = path.join(project_root_path, '.local_dir')
remote_dir = path.join(sep)    # vm root

upload_project(local_dir=local_dir, remote_dir=remote_dir, use_sudo=True)

this will fail because .local dir is hidden.

[localhost] local: vagrant status
[localhost] local: vagrant ssh-config
[localhost] local: tar -czf /var/folders/8z/dm68k1vd1zxf73slr540s3wc0000gn/T/tmpMGJuIH/local_dir.tar.gz -C /Users/aidanmelen/Sampleminded/infrastructure/python_tools/tastetest/test local_dir
tar: local_dir: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.

Fatal error: local() encountered an error (return code 1) while executing 'tar -czf /var/folders/8z/dm68k1vd1zxf73slr540s3wc0000gn/T/tmpMGJuIH/local_dir.tar.gz -C /Users/aidanmelen/Sampleminded/infrastructure/python_tools/tastetest/test local_dir'

Aborting.
[localhost] local: rm -rf /var/folders/8z/dm68k1vd1zxf73slr540s3wc0000gn/T/tmpMGJuIH

it will work properly if '.local_dir; is not hidden, such as 'local_dir'

@aidanmelen aidanmelen closed this Mar 4, 2017

@aidanmelen aidanmelen reopened this Mar 4, 2017

@ploxiln

This comment has been minimized.

ploxiln commented Mar 4, 2017

It looks like you specify "local_dir" instead of the actual name of the folder ".local_dir" e.g.

local_dir = path.join(project_root_path, 'local_dir')

should be

local_dir = path.join(project_root_path, '.local_dir')

@aidanmelen

This comment has been minimized.

aidanmelen commented Mar 4, 2017

(update) Well yes, that's true, Using 'local_dir' will pass. The problem is that upload_project cannot handle hidden directories. For this reason, '.local_dir would fail.

@ploxiln

This comment has been minimized.

ploxiln commented Mar 4, 2017

Seems to handle hidden directories for me:

import os

from fabric.api import task, run
from fabric.contrib.project import upload_project

@task
def test():
    local_dir = os.path.abspath(".local_dir")
    upload_project(local_dir=local_dir, remote_dir="/tmp", use_sudo=True)
$ fab --version
Fabric 1.13.1
Paramiko 2.1.1
$ ls -la .local_dir
total 16
drwxr-xr-x   4 pierce  staff  136 Mar  4 12:38 .
drwxr-xr-x  12 pierce  staff  408 Mar  4 12:44 ..
-rw-r--r--   1 pierce  staff   52 Mar  4 12:38 testfile.log
-rw-r--r--   1 pierce  staff   11 Mar  4 12:38 testfile.txt
$ fab --show=debug -H testdeploy03.ec2.st-av.net test
Using fabfile '/Users/pierce/scratch/fabwork/fabfile.py'
Commands to run: test
Parallel tasks now using pool size of 1
[testdeploy03.ec2.st-av.net] Executing task 'test'
[localhost] local: tar -czf /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmpOMtLKB/.local_dir.tar.gz -C /Users/pierce/scratch/fabwork .local_dir
[testdeploy03.ec2.st-av.net] put: /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmpOMtLKB/.local_dir.tar.gz -> /tmp/.local_dir.tar.gz
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "mv \"7b854f99c175b8342707837da591666e8dcea132\" \"/tmp/.local_dir.tar.gz\""
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "cd /tmp >/dev/null && tar -xzf .local_dir.tar.gz"
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "cd /tmp >/dev/null && rm -f .local_dir.tar.gz"
[localhost] local: rm -rf /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmpOMtLKB

Done.
Disconnecting from testdeploy03.ec2.st-av.net... done.

upload_project() does have the strange problem with needing local_dir to contain a path separator in it (so making it an absolute path works, which is equivalent to prepending with getcwd() as you did). But this function, along with most of "contrib", will most likely be dropped in fabric 2.0.

@aidanmelen

This comment has been minimized.

aidanmelen commented Mar 4, 2017

Gotcha @ploxiln, I didn't realize that upload_project() required absolute paths for arguments. I will give that a try.

@ploxiln

This comment has been minimized.

ploxiln commented Mar 4, 2017

In your original report, you did specify an absolute path, but you did not specify the correct folder name.

In fact, an absolute path is not needed, just any path with more than a single component (meaning, with a slash). This works:

from fabric.api import env, task, run, puts
from fabric.contrib.project import upload_project

@task
def test():
    upload_project("./.local_dir", "/tmp", use_sudo=True)
$ fab --show=debug -H testdeploy03.ec2.st-av.net test
Using fabfile '/Users/pierce/scratch/fabwork/fabfile.py'
Commands to run: test
Parallel tasks now using pool size of 1
[testdeploy03.ec2.st-av.net] Executing task 'test'
[localhost] local: tar -czf /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmp1mfo7q/.local_dir.tar.gz -C . .local_dir
[testdeploy03.ec2.st-av.net] put: /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmp1mfo7q/.local_dir.tar.gz -> /tmp/.local_dir.tar.gz
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "mv \"7b854f99c175b8342707837da591666e8dcea132\" \"/tmp/.local_dir.tar.gz\""
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "cd /tmp >/dev/null && tar -xzf .local_dir.tar.gz"
[testdeploy03.ec2.st-av.net] sudo: sudo -S -p 'sudo password:'  /bin/bash -l -c "cd /tmp >/dev/null && rm -f .local_dir.tar.gz"
[localhost] local: rm -rf /var/folders/ny/tsl8z9qx453630cxnzw0_dj80000gn/T/tmp1mfo7q
@aidanmelen

This comment has been minimized.

aidanmelen commented Mar 4, 2017

okay thanks for your input. I think this issue can be closed.

@aidanmelen aidanmelen closed this Mar 4, 2017

ploxiln added a commit to ploxiln/fab-classic that referenced this issue Apr 3, 2017

ploxiln added a commit to ploxiln/fab-classic that referenced this issue Apr 3, 2017

ploxiln added a commit to ploxiln/fab-classic that referenced this issue Apr 3, 2017

ploxiln added a commit to ploxiln/fab-classic that referenced this issue Apr 3, 2017

ploxiln added a commit to ploxiln/fab-classic that referenced this issue Apr 3, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment