Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

datalad get over ssh causes (again) multiple ssh password prompts #2988

Open
yarikoptic opened this issue Nov 8, 2018 · 8 comments
Open

datalad get over ssh causes (again) multiple ssh password prompts #2988

yarikoptic opened this issue Nov 8, 2018 · 8 comments

Comments

@yarikoptic
Copy link
Member

@yarikoptic yarikoptic commented Nov 8, 2018

ran into on discovery with annex 6.20180926-gc906aaf -- I thought that problem was fixed, need to doublecheck/verify

@yarikoptic

This comment has been minimized.

Copy link
Member Author

@yarikoptic yarikoptic commented Nov 8, 2018

seems to be a regression indeed:

git-annex version: 6.20180926-gc906aaf
...
[yhalchen@discovery7 QA]$ datalad --version
datalad 0.10.3.1

I will check with the most recent git annex build I need to do anyways.

another even nastier side effect -- if it is a new ssh host -- I would get many prompts for confirming its ssh fingerprint

@bpoldrack

This comment has been minimized.

Copy link
Member

@bpoldrack bpoldrack commented Nov 8, 2018

Generally speaking: It was fixed. I don't use key authentication everywhere, so I should have ran into it, if it wasn't fixed.

Can you show it in more detail? Is it really an annex call that prompts for passwd? Could be git as well. The connection sharing relies on how to tell annex and git how to use ssh. If the way of configuring that has changed with one of them, we might need to adapt.

@yarikoptic

This comment has been minimized.

Copy link
Member Author

@yarikoptic yarikoptic commented Nov 8, 2018

nope -- annex:

[yhalchen@discovery7 QA]$ datalad --version
datalad 0.10.3.1
[yhalchen@discovery7 QA]$ git annex version | head
git-annex version: 6.20180926-gc906aaf
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.17.1 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.7.0 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
operating system: linux x86_64
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
local repository version: 5
[yhalchen@discovery7 QA]$ git annex get -J2 sub-qa/
get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2Xs4_dwi.nii.gz get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2_dwi.nii.gz (from origin...) 
(from origin...) 
yoh@falkor.datalad.org's password: yoh@falkor.datalad.org's password: 

from --debug:

get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2_dwi.nii.gz get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2Xs4_dwi.nii.gz [2018-11-08 12:32:52.386649] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2018-11-08 12:32:52.386671] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2018-11-08 12:32:52.387506] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2018-11-08 12:32:52.388247] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
(from origin...) 
(from origin...) 
[2018-11-08 12:32:52.419763] chat: ssh ["yoh@falkor.datalad.org","-T","git-annex-shell 'p2pstdio' '/srv/datasets.datalad.org/www/dbic/QA/' '--debug' 'd4b0e10b-1fd9-4345-b2e0-87744c0594ea' --uuid 7d9ed214-3e5f-4cc8-ac88-f397145b2d4c"]
[2018-11-08 12:32:52.419887] chat: ssh ["yoh@falkor.datalad.org","-T","git-annex-shell 'p2pstdio' '/srv/datasets.datalad.org/www/dbic/QA/' '--debug' 'd4b0e10b-1fd9-4345-b2e0-87744c0594ea' --uuid 7d9ed214-3e5f-4cc8-ac88-f397145b2d4c"]
yoh@falkor.datalad.org's password: yoh@falkor.datalad.org's password: 
@yarikoptic

This comment has been minimized.

Copy link
Member Author

@yarikoptic yarikoptic commented Nov 8, 2018

situation is worse with fresh build, although I think I saw this messages after interruption of previous attempt before as well. Here is a clean comparison:

recentish version -- double password prompt

[yhalchen@discovery7 QA]$ git annex version | head -n1 ; git annex get -J2 sub-qa/
git-annex version: 6.20180926-gc906aaf
get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2Xs4_dwi.nii.gz get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2_dwi.nii.gz (from origin...) 
(from origin...) 
yoh@falkor.datalad.org's password: yoh@falkor.datalad.org's password: 

bleeding edge:

[yhalchen@discovery7 QA]$ export PATH=/ihome/yhalchen/7.20181105+git22-g4c7236c58/usr/lib/git-annex.linux:$PATH
[yhalchen@discovery7 QA]$ git annex version | head -n1 ; git annex get -J2 sub-qa/
git-annex version: 7.20181105+git22-g4c7236c58-1~ndall+1
get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2Xs4_dwi.nii.gz get sub-qa/ses-20161128/dwi/sub-qa_ses-20161128_acq-DTIX30Xp2_dwi.nii.gz (transfer already in progress, or unable to take transfer lock) 
  Unable to access these remotes: origin

  Try making some of these repositories available:
  	6384a551-a41d-4290-b186-9258befede97 -- bids@rolando:/inbox/BIDS/dbic/QA
   	7d9ed214-3e5f-4cc8-ac88-f397145b2d4c -- yoh@falkor:/srv/datasets.datalad.org/www/dbic/QA [origin]
   	ba8f2cea-f229-422c-82be-6580e5e07ed5 -- yoh@smaug:/mnt/datasets/datalad/crawl/dbic/QA
failed
get sub-qa/ses-20161128/func/sub-qa_ses-20161128_task-rest_acq-p2Xs4X35mm_bold.nii.gz (from origin...) 
(from origin...) 


  thread blocked indefinitely in an MVar operation
  thread blocked indefinitely in an STM transaction

  Unable to access these remotes: origin

  Unable to access these remotes: origin

  Try making some of these repositories available:
  	6384a551-a41d-4290-b186-9258befede97 -- bids@rolando:/inbox/BIDS/dbic/QA
   	7d9ed214-3e5f-4cc8-ac88-f397145b2d4c -- yoh@falkor:/srv/datasets.datalad.org/www/dbic/QA [origin]
   	ba8f2cea-f229-422c-82be-6580e5e07ed5 -- yoh@smaug:/mnt/datasets/datalad/crawl/dbic/QA
failed

  Try making some of these repositories available:
  	6384a551-a41d-4290-b186-9258befede97 -- bids@rolando:/inbox/BIDS/dbic/QA
   	7d9ed214-3e5f-4cc8-ac88-f397145b2d4c -- yoh@falkor:/srv/datasets.datalad.org/www/dbic/QA [origin]
   	ba8f2cea-f229-422c-82be-6580e5e07ed5 -- yoh@smaug:/mnt/datasets/datalad/crawl/dbic/QA
failed
git-annex: thread blocked indefinitely in an STM transaction

note that those are two different builds of annex - recentish one is the "stock build" and the bleeding edge is my neurodebian build on buster -- may be it relates (different libraries used etc )

@bpoldrack

This comment has been minimized.

Copy link
Member

@bpoldrack bpoldrack commented Nov 8, 2018

man git-annex

GIT_SSH, GIT_SSH_COMMAND
Handled similarly to the same as described in git(1). The one difference is that git-
annex will sometimes pass an additional "-n" parameter to these, as the first parameter,
to prevent ssh from reading from stdin. Since that can break existing uses of these
environment variables that don't expect the extra parameter, you will need to set
GIT_ANNEX_USE_GIT_SSH=1 to make git-annex support these.

          Note that setting either of these environment variables prevents git-annex from automat‐
          ically enabling ssh connection caching (see annex.sshcaching), so it will slow down some
          operations with remotes over ssh. It's up to you to enable ssh connection caching if you
          need it; see ssh's documentation.

          Also,  annex.ssh-options  and remote.<name>.annex-ssh-options won't have any effect when
          these envionment variables are set.

          Usually it's better to configure any desired options through your ~/.ssh/config file, or
          by setting annex.ssh-options.

I suspect that this might have changed. As far as I remember we used both remote.<name>.annex-ssh-options for annex AND the GIT_SSH_COMMAND for git calls. If the consideration of the env var has changed by annex, this might mess up our setup.

@yarikoptic

This comment has been minimized.

Copy link
Member Author

@yarikoptic yarikoptic commented Nov 8, 2018

FWIW, nothing changes in the behavior I observe by setting GIT_ANNEX_USE_GIT_SSH=1

@yarikoptic

This comment has been minimized.

Copy link
Member Author

@yarikoptic yarikoptic commented Nov 9, 2018

reported to @joeyh : http://git-annex.branchable.com/bugs/multiple_ssh_prompts__44___and_thread_blocked_indefinitely_in_an___63____63____63___transaction/ with hope for a quick resolution

On a related note - I will see if I could establish buildbot presence on one of those boxes to verify correct operation, meanwhile will run tests manually to see where we stand.

@yarikoptic

This comment has been minimized.

Copy link
Member Author

@yarikoptic yarikoptic commented Nov 9, 2018

not my luck

(dev) [yhalchen@discovery datalad]$ python -m nose -s -v datalad/
Segmentation fault (core dumped)
(dev) [yhalchen@discovery datalad]$ python -c 'import datalad.customremotes.archives'
Segmentation fault (core dumped)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.