transfers.rsync ignores ssh agent and always asks for key password

[max-arnold]

Thanks for making patchwork compatible with Python 3! Trying to port one small project to Fabric v2:

from invoke import task
from patchwork.transfers import rsync

def get_proj_path():
    from fabric.main import program
    return program.collection.loaded_from

@task
def upload(c):
    rsync(c, '%s/configs' % get_proj_path(), '/tmp')

> /Users/user/.virtualenvs/test1/lib/python3.6/site-packages/patchwork/transfers.py(131)rsync()
    129     cmd = cmd.format(options, source, user, host, target)
    130     import ipdb; ipdb.set_trace()
--> 131     return c.local(cmd)

Remote command works fine with ssh agent running:

ipdb> print(c.run('pwd'))
/root
Command exited with status 0.
=== stdout ===
/root

(no stderr)

c.local("rsync -pthrvz --rsh='ssh -p 22 ' /Users/user/work/project/configs root@1.2.3.4:/tmp") asks for ssh key:

ipdb> c
Enter passphrase for key '/Users/user/.ssh/id_rsa':

Running the same command from the shell works fine:

% rsync  -pthrvz  --rsh='ssh  -p 22 ' /Users/user/work/project/configs root@1.2.3.4:/tmp
sending an incremental file list

sent 237 bytes  received 18 bytes  72.86 bytes/sec
total size is 10.45K  speedup is 40.99

[max-arnold]

The reason seems to be replace_env = True in the fabric/config.py:

        # TODO: as to that, this is a core problem, Fabric wants split
        # local/remote stuff, eg replace_env wants to be False for local and
        # True remotely; shell wants to differ depending on target (and either
        # way, does not want to use local interrogation for remote)
        # TODO: is it worth moving all of our 'new' settings to a discrete
        # namespace for cleanliness' sake? e.g. ssh.port, ssh.user etc.
        # It wouldn't actually simplify this code any, but it would make it
        # easier for users to determine what came from which library/repo.
        defaults = InvokeConfig.global_defaults()
        ours = {
            # New settings
            'connect_kwargs': {},
            'forward_agent': False,
            'gateway': None,
            'load_ssh_configs': True,
            'port': 22,
            'run': {
                'replace_env': True,
            },
            'runners': {
                'remote': Remote,
            },

But I have no idea why it applies to local(). Any workarounds?

[bitprophet]

Probably has to do with the currently-too-thin boundary between run and local, there are outstanding issues there that need solving and this feels like an example of one of them.

Eg I think what needs to happen is vanilla-Invoke's run() wants to honor config.run.* but Fabric 2 wants to clone config.run.* to be config.local.* so that the two can have differing defaults. That's a Fabric, not Patchwork, level change - to be clear.

I'm still trying to get Patchwork and Invocations both up to snuff re: being publication and pull request worthy, after that I'll get back to this sort of feature related work. PyCon sprints!

[max-arnold]

Below is my current workaround for this issue:

def rsync_with_agent(ctx, *args, **kwargs):
    """Ugly workaround for https://github.com/fabric/patchwork/issues/16."""
    ssh_agent = os.environ.get('SSH_AUTH_SOCK', None)
    if ssh_agent:
        ctx.config['run']['env']['SSH_AUTH_SOCK'] = ssh_agent
    return rsync(ctx, *args, **kwargs)

[sheluchin]

I ran into the same issue and @max-arnold's workaround worked for me.

[bitprophet]

This never got linked to fabric/fabric#1752 - once that is solved this may go away or at least be easier to fix.

[dherykw]

I am not able to use this workaround, still needing the password

[krupan]

I'm having another problem related to this (problems with local mentioned above), trying to run my django unit tests before deploying:

c.local('poetry run python manage.py test')

Gives me this:

[KeyError]  
'PATH' 

But running remotely with c.run works just fine.