datalad doesn't fetch data

[ruwang]

I followed the slide here [http://nipy.org/heudiconv/#1], step by step. When I got to slide 3 and did the command datalad get -J6 YAROSLAV_DBIC-TEST1/, it outputted error:

(neuro) root@394c951177fa:/dicoms/PHANTOM1_3# datalad get -J6 YAROSLAV_DBIC-TEST1/
[ERROR  ] unknown argument: -J6 
usage: datalad get [-h] [-s LABEL] [-d PATH] [-r] [--recursion-limit LEVELS]
                   [-n] [-D DESCRIPTION] [--reckless] [-J NJOBS] [-v]
                   [PATH [PATH ...]]

Then I added a space after -j, and reran the command. It outputted the following error:

(neuro) root@394c951177fa:/dicoms/PHANTOM1_3# datalad get -J 6 YAROSLAV_DBIC-TEST1/
Total (0 ok, 547 failed out of 547):   0%|                                                                                                                   | 0.00/135M [00:03<?, ?B/s][WARNING] Running get resulted in stderr output: git-annex: get: 547 failed
 
                                                                                                                                                                                                   [ERROR  ] not available; Try making some of these repositories available:; 	14228697-3986-48c3-ba48-8718c7ec336a -- yoh@smaug:/mnt/datasets/datalad/crawl/dicoms/dartmouth-phantoms/PHANTOM1_3;  	28468356-71a3-4b45-b2ff-9d2c6f88748d -- yoh@hopa:~/picts/mris/PHANTOM1_3;  	869bee6b-7aae-4bb6-99aa-1884fb38f9dd -- yoh@falkor:/srv/datasets.datalad.org/www/test/dartmouth-siemens;  c38a9393-0086-4545-b725-b4d16367790b -- /srv/datasets.datalad.org/www/dicoms/dartmouth-phantoms/PHANTOM1_3; ; (Note that these git remotes have annex-ignore set: origin) [get(/dicoms/PHANTOM1_3/YAROSLAV_DBIC-TEST1/HEAD_ADVANCED_APPLICATIONS_LIBRARIES_20160824_104430_780000/AAHEAD_SCOUT_0001/YAROSLAV.MR.HEAD_ADVANCED_APPLICATIONS_LIBRARIES.0001.0006.2016.08.24.11.53.51.544091.5158050.IMA)] 

[lots of errors not shown here]

Any help here?

[yarikoptic]

space issue is resolved in the recent 0.10.3 AFAIK

otherwise looks like you lack any git remote to point to the falkor repository from which you should have got as "origin" when you did git clone http://datasets.datalad.org/dicoms/dartmouth-phantoms/PHANTOM1_3/.git:

$> git clone http://datasets.datalad.org/dicoms/dartmouth-phantoms/PHANTOM1_3/.git
Cloning into 'PHANTOM1_3'...

$> cd PHANTOM1_3 
YAROSLAV_DBIC-TEST1/

$> datalad get -J 6 YAROSLAV_DBIC-TEST1 
Total (71 ok out of 547):   9%|█▏            | 11.5M/135M [00:02<00:20, 5.99MB/s]ERROR:                                                                           
Interrupted by user while doing magic: KeyboardInterrupt() [cmd.py:_process_one_line:348]

$> git remote
origin

$> datalad siblings
.: here(+) [git]
.: origin(+) [http://datasets.datalad.org/dicoms/dartmouth-phantoms/PHANTOM1_3/.git (git)]

[ruwang]

Hi Yaroslav, thanks for the response. I have used the latest version. I just started learning heudiconv and datalad, and tried to get a working example. Appreciate your help.

$ docker pull nipy/heudiconv:latest
latest: Pulling from nipy/heudiconv
Digest: sha256:2ab866fca794794ccb80bbdf631a7a7e9df8eae85e8ee98f3ec5e1453edc01b8
Status: Image is up to date for nipy/heudiconv:latest
$ docker run -it --rm -v ~/dicom-test:/dicoms --entrypoint=bash nipy/heudiconv:latest
root@f16a26fcc2f3:/tmp# source activate neuro && cd /dicoms
(neuro) root@f16a26fcc2f3:/dicoms# git clone http://datasets.datalad.org/dicoms/dartmouth-phantoms/PHANTOM1_3/.git
Cloning into 'PHANTOM1_3'...
Checking connectivity: 16750, done.
Checking out files: 100% (551/551), done.
(neuro) root@f16a26fcc2f3:/dicoms# cd PHANTOM1_3/
(neuro) root@f16a26fcc2f3:/dicoms/PHANTOM1_3# git remote
origin
(neuro) root@f16a26fcc2f3:/dicoms/PHANTOM1_3# ls
YAROSLAV_DBIC-TEST1
(neuro) root@f16a26fcc2f3:/dicoms/PHANTOM1_3# datalad siblings
.: here(+) [git]
.: origin(-) [http://datasets.datalad.org/dicoms/dartmouth-phantoms/PHANTOM1_3/.git (git)]
(neuro) root@f16a26fcc2f3:/dicoms/PHANTOM1_3# datalad get -J 6 YAROSLAV_DBIC-TEST1
Total (0 ok, 547 failed out of 547):   0%|                                                                                                            | 0.00/135M [00:03<?, ?B/s][WARNING] Running get resulted in stderr output: git-annex: get: 547 failed
 
                                                                                                                                                                                            [ERROR  ] not available; Try making some of these repositories available:; 	14228697-3986-48c3-ba48-8718c7ec336a -- yoh@smaug:/mnt/datasets/datalad/crawl/dicoms/dartmouth-phantoms/PHANTOM1_3;  	28468356-71a3-4b45-b2ff-9d2c6f88748d -- yoh@hopa:~/picts/mris/PHANTOM1_3;  	869bee6b-7aae-4bb6-99aa-1884fb38f9dd -- yoh@falkor:/srv/datasets.datalad.org/www/test/dartmouth-siemens;  	c38a9393-0086-4545-b725-b4d16367790b -- /srv/datasets.datalad.org/www/dicoms/dartmouth-phantoms/PHANTOM1_3; ; (Note that these git remotes have annex-ignore set: origin) [get(/dicoms/PHANTOM1_3/YAROSLAV_DBIC-TEST1/HEAD_ADVANCED_APPLICATIONS_LIBRARIES_20160824_104430_780000/AAHEAD_SCOUT_0001/YAROSLAV.MR.HEAD_ADVANCED_APPLICATIONS_LIBRARIES.0001.0006.2016.08.24.11.53.51.544091.5158050.IMA)] 

[yarikoptic]

sorry I "doubted" you @ruwang... smells like some kind of an issue indeed, looking into it/replicating, thanks for the report!

[ruwang]

No worry at all @yarikoptic . I should have provided more info in my first post. Thank you for looking into this!

[yarikoptic]

oh jesus - something new (haven't seen this error report from git-annex before):

(neuro) root@1892bcc7f1e4:/dicoms/PHANTOM1_3# git annex get YAROSLAV_DBIC-TEST1/HEAD_ADVANCED_APPLICATIONS_LIBRARIES_20160824_104430_780000/AAHEAD_SCOUT_0001/YAROSLAV.MR.HEAD_ADVANCED_APPLICATIONS_LIBRARIES.0001.0001.2016.08.24.11.53.51.544091.5157975.IMA
(merging origin/git-annex into git-annex...)
(recording state in git...)
get YAROSLAV_DBIC-TEST1/HEAD_ADVANCED_APPLICATIONS_LIBRARIES_20160824_104430_780000/AAHEAD_SCOUT_0001/YAROSLAV.MR.HEAD_ADVANCED_APPLICATIONS_LIBRARIES.0001.0001.2016.08.24.11.53.51.544091.5157975.IMA download failed: ConnectionFailure Network.BSD.getProtocolByName: does not exist (no such protocol name: tcp)

  Remote origin not usable by git-annex; setting annex-ignore
(not available) 

happens with both installed there (6.20180626+gitg12cd64369-1~ndall+1) and updated (6.20180913+git33-g2cd5a723f-1~ndall+1) git annex. Smart people instructed to install netbase , which seems to resolve the issue. So just remove that initial clone (will make life easier, otherwise need to edit .git/config to remove annex ignore for origin), apt-get install netbase within the container, and try again -- must work

I will add netbase as run time dependency for git-annex-standalone pkg (FYI @joeyh).

[ruwang]

After installing the netbase, I tried skipping the git clone step and did the datalad get, it complained about path not associated with any dataset:

(neuro) root@eef601716774:/dicoms# datalad get -J 6 YAROSLAV_DBIC-TEST1/
[WARNING] path not associated with any dataset [get(/dicoms/YAROSLAV_DBIC-TEST1)] 
get(impossible): /dicoms/YAROSLAV_DBIC-TEST1 [path not associated with any dataset]
(neuro) root@eef601716774:/dicoms# datalad get -J 6 PHANTOM1_3/YAROSLAV_DBIC-TEST1/
[WARNING] path not associated with any dataset [get(/dicoms/PHANTOM1_3/YAROSLAV_DBIC-TEST1)] 
get(impossible): /dicoms/PHANTOM1_3/YAROSLAV_DBIC-TEST1 [path not associated with any dataset]

However, if I kept the git clone and updated the .git/config to remove the annex-ignore line, datalad get ran successfully!

Thanks @yarikoptic for your help!