ssh interactions are broken with fresh annex 6.20170403+gitgdbe23d695-1~ndall+1 #1456

yarikoptic · 2017-04-06T22:56:46Z

which I have managed to upload although I thought I have tested it locally first

I think (actually now certain since succeeds with 6.20170317+gitg8cd473c71-1~ndall+1 and fails with the 6.20170317+gitgfaecd73f3-1~ndall+1) it is due to

commit faecd73f327bee5067998dd6d398033fb09a26a6
Author: Joey Hess <joeyh@joeyh.name>
Date:   Fri Mar 17 16:02:47 2017 -0400

    Support GIT_SSH and GIT_SSH_COMMAND
    
    They are handled close the same as they are by git. However, unlike git,
    git-annex sometimes needs to pass the -n parameter when using these.
    
    So, this has the potential for breaking some setup, and perhaps there ought
    to be a ANNEX_USE_GIT_SSH=1 needed to use these. But I'd rather avoid that
    if possible, so let's see if anyone complains.
    
    Almost all places where "ssh" was run have been changed to support the env
    vars. Anything still calling sshOptions does not support them. In
    particular, rsync special remotes don't. Seems that annex-rsync-transport
    already gives sufficient control there.
    
    (Fixed in passing: Remote.Helper.Ssh.toRepo used to extract
    remoteAnnexSshOptions and pass them to sshOptions, which was redundant
    since sshOptions also extracts those.)
    
    This commit was sponsored by Jeff Goeke-Smith on Patreon.

~~and I guess we should just move from passing them via -c remote.*.annex-ssh-options~~

if that is so, following the commit msg -- @joeyh know that you did break someone's workflow and I am complaining! The issue is that we can't just use env variables to point to the socket paths (e.g. during sync), since those would be different for different remotes and multiple remotes might be communicated in a single git annex call.
@joeyh - please advise on how to proceed

The text was updated successfully, but these errors were encountered:

joeyh · 2017-04-07T00:20:59Z

I'm not clear how faecd73f327bee5067998dd6d398033fb09a26a6 broke datalad. Is datalad setting GIT_SSH/GIT_SSH_COMMAND when it runs git-annex?

…

-- see shy jo

yarikoptic · 2017-04-07T02:02:06Z

yeap. We set 'GIT_SSH_COMMAND' to datalad sshrun (if not already in the environment)

joeyh · 2017-04-07T15:09:24Z

Yaroslav Halchenko wrote:

yeap. We set 'GIT_SSH_COMMAND' to datalad sshrun (if not already in the environment)

And your argument parser for datalad sshrun does fall over when passed an extra -n as the first parameter? If so, I could implement the ANNEX_USE_GIT_SSH=1 thing. At least until I can find a way to not need to pass -n when running ssh.

…

-- see shy jo

joeyh · 2017-04-07T15:17:35Z

Verified in the source to SSHRun. Of course, you could make that code support the extra leading -n parmater and pass it through to ssh. This would fix the breakage too. Seems to me that datalad might want to make its sshrun be used when git-annex runs ssh. I see some rather ugly looking code in datalad that uses annex-ssh-options to make it use the shared connections now, which you would be able to get rid of if you used git-annex's GIT_SSH_COMMAND support.

…

-- see shy jo

yarikoptic · 2017-04-07T15:38:37Z

@joeyh yeah -- we didn't support -n, added it but it didn't fix the issue(s)... looking into it -- there might be more to the 'story' on our end (#1458) but still -- somehow it worked before and no longer does :-/

yeap -- we were passing via annex-ssh-options since that was the only way AFAIK to reuse the same connection (managed by us) by both annex and datalad... also we did use GIT_SSH_COMMAND to point to our ssh runner already for pure git interactions. Or what do you mean by " if you used git-annex's GIT_SSH_COMMAND support."? that if with new functionality in annex we just remove passing control paths via annex-ssh-options?

joeyh · 2017-04-07T15:39:19Z

Fixed on git-annex's side in b6f26bac86e1332410122d3dc43d9723e1ea8afa I'd still suggest making datalad support git-annex's variation of GIT_SSH_COMMAND and use it instead of annex-ssh-options.

…

-- see shy jo

yarikoptic · 2017-04-07T15:43:54Z

thanks -- I will check it out right away

joeyh · 2017-04-07T15:54:10Z

Yaroslav Halchenko wrote:

Or what do you mean by " if you used git-annex's GIT_SSH_COMMAND support."? that if with new functionality in annex we just remove passing control paths via annex-ssh-options?

Correct, you could.

…

-- see shy jo

yarikoptic · 2017-04-07T16:10:59Z

b6f26bac86e1332410122d3dc43d9723e1ea8afa FTBFS due to failing tests for me (in stretch):

...
    move (ssh remote):                                    On branch master
nothing to commit, working tree clean
ssh: connect to host localhost port 22: Connection refused
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(235) [Receiver=3.1.2]
FAIL (0.22s)
      move --from of file failed
    copy:                                                 On branch master
nothing to commit, working tree clean
...
8 out of 293 tests failed (224.52s)

on our end -- fixed up for #1458 and -n handling but it didn't resolve our failing tests... (at least not all of them -- re-running as of speaking)

yarikoptic · 2017-04-07T20:14:43Z

@joeyh part (or whole?) of the issue is quite elaborate non-escaping or just concatenation of strings in now invoked 'configlist' (didn't invoke before for annex sync I believe, or at least didn't for us in that test) command:

$> ps w --pid 31252         
  PID TTY      STAT   TIME COMMAND
31252 pts/6    R+     2:51 /home/yoh/proj/datalad/datalad/venvs/dev/bin/python2 /home/yoh/proj/datalad/datalad/venvs/dev/bin/datalad sshrun datalad-test 'git-annex-shell '"'"'configlist'"'"' '"'"'/tmp/datalad_temp_testrepo_dPT_tB'"'"' '"'"'--'"'"' '"'"'autoinit=1'"'"' '"'"'--'"'"''

$> ps w --pid 31214
  PID TTY      STAT   TIME COMMAND
31214 pts/6    S+     0:00 /usr/lib/git-annex.linux/exe/sh --library-path /usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit:/usr/lib/git-annex.linux//etc/ld.so.conf.d:/usr/lib/git-annex.linux//lib64:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu:/usr

$> ps w --pid 31214 | cat
  PID TTY      STAT   TIME COMMAND
31214 pts/6    S+     0:00 /usr/lib/git-annex.linux/exe/sh --library-path /usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit:/usr/lib/git-annex.linux//etc/ld.so.conf.d:/usr/lib/git-annex.linux//lib64:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu:/usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/sh/sh -n -c datalad sshrun "$@" datalad sshrun datalad-test 'git-annex-shell '"'"'configlist'"'"' '"'"'/tmp/datalad_temp_testrepo_dPT_tB'"'"' '"'"'--'"'"' '"'"'autoinit=1'"'"' '"'"'--'"'"''

$> ps w --pid 31215 | cat
  PID TTY      STAT   TIME COMMAND
31215 pts/6    S+     0:00 /home/yoh/proj/datalad/datalad/venvs/dev/bin/python2 /home/yoh/proj/datalad/datalad/venvs/dev/bin/datalad sshrun datalad-test 'git-annex-shell '"'"'configlist'"'"' '"'"'/tmp/datalad_temp_testrepo_dPT_tB'"'"' '"'"'--'"'"' '"'"'autoinit=1'"'"' '"'"'--'"'"''

$> ps w --pid 31252 | cat
  PID TTY      STAT   TIME COMMAND
31252 pts/6    R+     3:13 /home/yoh/proj/datalad/datalad/venvs/dev/bin/python2 /home/yoh/proj/datalad/datalad/venvs/dev/bin/datalad sshrun datalad-test 'git-annex-shell '"'"'configlist'"'"' '"'"'/tmp/datalad_temp_testrepo_dPT_tB'"'"' '"'"'--'"'"' '"'"'autoinit=1'"'"' '"'"'--'"'"''

$> cat /proc/31252/cmdline
/home/yoh/proj/datalad/datalad/venvs/dev/bin/python2/home/yoh/proj/datalad/datalad/venvs/dev/bin/dataladsshrundatalad-test'git-annex-shell '"'"'configlist'"'"' '"'"'/tmp/datalad_temp_testrepo_dPT_tB'"'"' '"'"'--'"'"' '"'"'autoinit=1'"'"' '"'"'--'"'"''%

and then our sshrun somehow fails to run that ;-)

yarikoptic · 2017-04-07T20:18:12Z

@joeyh here is the comparison from running "good/bad" versions of annex (grepping on the path of the socket which we had in each of those relevant invocations):

$> grep -e Run.*ce7ee7ea log-output.good 
{4146}[DEBUG] Running: ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'sync', '--debug', '-c', 'remote.ssh-remote-1.annex-ssh-options=-o ControlMaster=auto -S /home/yoh/.cache/datalad/sockets/ce7ee7ea']
{10990}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{10990}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{10990}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', "sh -e -c 'dirname $(readlink -f $(which git-annex-shell))'"]
{10990}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', 'export "PATH=/usr/lib/git-annex.linux:$PATH"; git-upload-pack \'/tmp/datalad_temp_testrepo_B9HaXK\'']
{11069}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{11069}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{11069}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', "sh -e -c 'dirname $(readlink -f $(which git-annex-shell))'"]
{11069}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', 'export "PATH=/usr/lib/git-annex.linux:$PATH"; git-receive-pack \'/tmp/datalad_temp_testrepo_B9HaXK\'']
{11144}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{11144}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{11144}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', "sh -e -c 'dirname $(readlink -f $(which git-annex-shell))'"]
{11144}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', 'export "PATH=/usr/lib/git-annex.linux:$PATH"; git-receive-pack \'/tmp/datalad_temp_testrepo_B9HaXK\'']
{11230}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{11230}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{11230}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', "sh -e -c 'dirname $(readlink -f $(which git-annex-shell))'"]
{11230}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', 'export "PATH=/usr/lib/git-annex.linux:$PATH"; git-receive-pack \'/tmp/datalad_temp_testrepo_B9HaXK\'']
{4146}[DEBUG] Running: ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'sync', '--debug', '-c', 'remote.ssh-remote-1.annex-ssh-options=-o ControlMaster=auto -S /home/yoh/.cache/datalad/sockets/ce7ee7ea', '-c', 'remote.ssh-remote-2.annex-ssh-options=-o ControlMaster=auto -S /home/yoh/.cache/datalad/sockets/ef2d633e', 'ssh-remote-2']
{4146}[DEBUG] Running: ['ssh', '-O', 'stop', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']

$> grep -e Run.*ce7ee7ea log-output.bad 
{12163}[DEBUG] Running: ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'sync', '--debug', '-c', 'remote.ssh-remote-1.annex-ssh-options=-o ControlMaster=auto -S /home/yoh/.cache/datalad/sockets/ce7ee7ea']
{18716}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{18716}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', "sh -e -c 'dirname $(readlink -f $(which git-annex-shell))'"]
{18716}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', 'export "PATH=/usr/lib/git-annex.linux:$PATH"; \'git-annex-shell \'"\'"\'configlist\'"\'"\' \'"\'"\'/tmp/datalad_temp_testrepo_3dZQkL\'"\'"\' \'"\'"\'--\'"\'"\' \'"\'"\'autoinit=1\'"\'"\' \'"\'"\'--\'"\'"\'\'']
{18716}[DEBUG] Running: ['ssh', '-O', 'stop', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{18757}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{18757}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', "sh -e -c 'dirname $(readlink -f $(which git-annex-shell))'"]
{18757}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', 'export "PATH=/usr/lib/git-annex.linux:$PATH"; git-upload-pack \'/tmp/datalad_temp_testrepo_3dZQkL\'']
{18757}[DEBUG] Running: ['ssh', '-O', 'stop', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{18841}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{18841}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', "sh -e -c 'dirname $(readlink -f $(which git-annex-shell))'"]
{18841}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', 'export "PATH=/usr/lib/git-annex.linux:$PATH"; git-upload-pack \'/tmp/datalad_temp_testrepo_3dZQkL\'']
{18841}[DEBUG] Running: ['ssh', '-O', 'stop', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{18938}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{18938}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', "sh -e -c 'dirname $(readlink -f $(which git-annex-shell))'"]
{18938}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', 'export "PATH=/usr/lib/git-annex.linux:$PATH"; git-receive-pack \'/tmp/datalad_temp_testrepo_3dZQkL\'']
{18938}[DEBUG] Running: ['ssh', '-O', 'stop', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{19010}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{19010}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', "sh -e -c 'dirname $(readlink -f $(which git-annex-shell))'"]
{19010}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', 'export "PATH=/usr/lib/git-annex.linux:$PATH"; git-receive-pack \'/tmp/datalad_temp_testrepo_3dZQkL\'']
{19010}[DEBUG] Running: ['ssh', '-O', 'stop', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{19082}[DEBUG] Running: ['ssh', '-O', 'check', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']
{19082}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', "sh -e -c 'dirname $(readlink -f $(which git-annex-shell))'"]
{19082}[DEBUG] Running: ['ssh', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test', 'export "PATH=/usr/lib/git-annex.linux:$PATH"; git-receive-pack \'/tmp/datalad_temp_testrepo_3dZQkL\'']
{19082}[DEBUG] Running: ['ssh', '-O', 'stop', '-o', 'ControlPath="/home/yoh/.cache/datalad/sockets/ce7ee7ea"', 'datalad-test']

yarikoptic · 2017-04-07T20:36:33Z

@joeyh note how the entire call for git-annex-shell is also wrapped into a single string (surrounded by ') when it is already a single argument on a command line. not sure how that magic should work out since doesn't even work in basic shell invocation of ssh:

$> ssh localhost "'echo \"123\"'"
zsh:1: command not found: echo "123"
*$> ssh test@localhost "'echo \"123\"'"
bash: echo "123": command not found

(surrounded with "" to make it into a single command line argument for ssh invocation)

joeyh · 2017-04-07T21:14:06Z

It may be that git-annex's GIT_SSH_COMMAND or GIT_SSH emulation has a bug in it. If so, please file a bug report on git-annex with the details, including how to reproduce it etc.

…

-- see shy jo

joeyh · 2017-04-10T15:59:07Z

I looked at this again, and I still lack enough information to help with it. I lack even a clear description of what the problem is.

If the problem is git-annex test failing, then I can't reproduce that.

If the problem is something complicated involving GIT_SSH as used by datalad and shell escaping, I will need to know what environment variable is getting set to what.

yarikoptic · 2017-04-12T02:58:04Z

was almost fully addressed by #1461 and current annex version 6.20170408+gitg804f06baa-1~ndall+1, without explicit GIT_ANNEX_USE_GIT_SSH=1, should function normally according to the tests

yarikoptic added regression annex Git-annex related issue labels Apr 6, 2017

yarikoptic mentioned this issue Apr 7, 2017

BF: direct mode/v6 status PR against return value revolution #1450

Merged

10 tasks

yarikoptic mentioned this issue Apr 8, 2017

RF+ENH+BF: attempts to fix up for recent git-annex changes #1461

Merged

6 tasks

yarikoptic closed this as completed Apr 12, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ssh interactions are broken with fresh annex 6.20170403+gitgdbe23d695-1~ndall+1 #1456

ssh interactions are broken with fresh annex 6.20170403+gitgdbe23d695-1~ndall+1 #1456

yarikoptic commented Apr 6, 2017 •

edited

Loading

joeyh commented Apr 7, 2017 via email

yarikoptic commented Apr 7, 2017

joeyh commented Apr 7, 2017 via email

joeyh commented Apr 7, 2017 via email

yarikoptic commented Apr 7, 2017

joeyh commented Apr 7, 2017 via email

yarikoptic commented Apr 7, 2017

joeyh commented Apr 7, 2017 via email

yarikoptic commented Apr 7, 2017

yarikoptic commented Apr 7, 2017

yarikoptic commented Apr 7, 2017

yarikoptic commented Apr 7, 2017

joeyh commented Apr 7, 2017 via email

joeyh commented Apr 10, 2017

yarikoptic commented Apr 12, 2017

ssh interactions are broken with fresh annex 6.20170403+gitgdbe23d695-1~ndall+1 #1456

ssh interactions are broken with fresh annex 6.20170403+gitgdbe23d695-1~ndall+1 #1456

Comments

yarikoptic commented Apr 6, 2017 • edited Loading

joeyh commented Apr 7, 2017 via email

yarikoptic commented Apr 7, 2017

joeyh commented Apr 7, 2017 via email

joeyh commented Apr 7, 2017 via email

yarikoptic commented Apr 7, 2017

joeyh commented Apr 7, 2017 via email

yarikoptic commented Apr 7, 2017

joeyh commented Apr 7, 2017 via email

yarikoptic commented Apr 7, 2017

yarikoptic commented Apr 7, 2017

yarikoptic commented Apr 7, 2017

yarikoptic commented Apr 7, 2017

joeyh commented Apr 7, 2017 via email

joeyh commented Apr 10, 2017

yarikoptic commented Apr 12, 2017

yarikoptic commented Apr 6, 2017 •

edited

Loading