datalad install -r datasets -s /// all failed to operate correctly

[yarikoptic]

it did install all/openfmri (but reported installing each one twice, starting with ds000148) but for the others (crcns, etc) nothing was done and weird installation successes were reported:

> datalad install -r datasets -s /// all
...
2016-08-09 22:15:07,676 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/openfmri/ds000204> from http://datasets.datalad.org/openfmri/ds000204/.git (install.py:375)
2016-08-09 22:15:07,729 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/openfmri/ds000204> from http://datasets.datalad.org/openfmri/./ds000204/.git (install.py:375)
2016-08-09 22:15:07,787 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/openfmri/ds000203> from http://datasets.datalad.org/openfmri/ds000203/.git (install.py:375)
2016-08-09 22:15:07,840 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/openfmri/ds000203> from http://datasets.datalad.org/openfmri/./ds000203/.git (install.py:375)
2016-08-09 22:15:08,075 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/neurovault> from http://datasets.datalad.org/neurovault/.git (install.py:375)
2016-08-09 22:15:08,144 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/neurovault> from http://datasets.datalad.org/./neurovault/.git (install.py:375)
2016-08-09 22:15:08,215 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/crcns> from http://datasets.datalad.org/crcns/.git (install.py:375)
2016-08-09 22:15:08,280 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/crcns> from http://datasets.datalad.org/./crcns/.git (install.py:375)
2016-08-09 22:15:08,346 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/labs> from http://datasets.datalad.org/labs/.git (install.py:375)
2016-08-09 22:15:08,412 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/labs> from http://datasets.datalad.org/./labs/.git (install.py:375)
2016-08-09 22:15:08,485 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/indi> from http://datasets.datalad.org/indi/.git (install.py:375)
2016-08-09 22:15:08,551 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all/indi> from http://datasets.datalad.org/./indi/.git (install.py:375)
1 installed item is available at
<Dataset path=/mnt/datasets/datalad/test-crawled2/all>
datalad install -r datasets -s /// all  30.75s user 20.72s system 72% cpu 1:11.22 total

$> ls all
neurovault/  openfmri/
(venv-tests)2 10219.....................................:Tue 09 Aug 2016 10:42:51 PM EDT:.
smaug:/mnt/datasets/datalad/test-crawled2
$> ls all/neurovault
(venv-tests)2 10220.....................................:Tue 09 Aug 2016 10:42:57 PM EDT:.
smaug:/mnt/datasets/datalad/test-crawled2
$> ls all/openfmri
ds000001/  ds000005/  ds000007/  ds000011/  ds000051/  ds000101/  ds000105/  ds000109/  ds000113/   ds000114/  ds000116/  ds000140/
ds000003/  ds000006/  ds000009/  ds000017/  ds000052/  ds000102/  ds000107/  ds000110/  ds000113b/  ds000115/  ds000117/  ds000203/

and then attempt to install crcns "manually" lead to infinite recursion (until crashed)

$> datalad install -r datasets all/crcns
2016-08-09 22:43:32,235 [INFO   ] Installing /mnt/datasets/datalad/test-crawled2/all/crcns (install.py:375)
2016-08-09 22:43:32,302 [INFO   ] Installing /mnt/datasets/datalad/test-crawled2/all/crcns (install.py:375)
...

so there is some major difference in how later datasets got published I guess although I don't see yet obvious difference :-/

[yarikoptic]

actually even without recursion installation of those awkwardly reported installed (but not really installed) subdatasets resulted in infinite recursion:

$> datalad install openfmri/ds000113d
2016-08-09 23:04:27,290 [INFO   ] Installing /mnt/datasets/datalad/test-crawled2/all-nonrecursive/openfmri/ds000113d (install.py:375)
2016-08-09 23:04:27,481 [INFO   ] Installing /mnt/datasets/datalad/test-crawled2/all-nonrecursive/openfmri/ds000113d (install.py:375)
2016-08-09 23:04:27,612 [INFO   ] Installing /mnt/datasets/datalad/test-crawled2/all-nonrecursive/openfmri/ds000113d (install.py:375)
...

although installs fine individually:

$> datalad install ///openfmri/ds000113d
2016-08-09 23:05:31,940 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/ds000113d> from http://datasets.datalad.org/openfmri/ds000113d (install.py:375)
1 installed item is available at
<Dataset path=/mnt/datasets/datalad/test-crawled2/ds000113d>

[yarikoptic]

just notes to myself to prove my sanity:

$> datalad install --source /// all-nonrecursive
2016-08-10 09:54:53,922 [INFO   ] Installing /mnt/datasets/datalad/test-crawled2/all-nonrecursive from http://datasets.datalad.org/ (install.py:375)
1 installed item is available at
<Dataset path=/mnt/datasets/datalad/test-crawled2/all-nonrecursive>

$> datalad install all-nonrecursive/crcns
2016-08-10 09:55:09,529 [INFO   ] Installing /mnt/datasets/datalad/test-crawled2/all-nonrecursive/crcns (install.py:375)
2016-08-10 09:55:09,602 [INFO   ] Installing /mnt/datasets/datalad/test-crawled2/all-nonrecursive/crcns (install.py:375)
....

but then all of a sudden, after I added a lgr.debug call started to work

$> rm -rf all-nonrecursive; datalad install --source /// all-nonrecursive; datalad install all-nonrecursive/crcns
2016-08-10 10:02:48,877 [INFO   ] Installing /mnt/datasets/datalad/test-crawled2/all-nonrecursive from http://datasets.datalad.org/ (install.py:378)
1 installed item is available at
<Dataset path=/mnt/datasets/datalad/test-crawled2/all-nonrecursive>
2016-08-10 10:02:49,439 [INFO   ] Installing /mnt/datasets/datalad/test-crawled2/all-nonrecursive/crcns (install.py:378)
2016-08-10 10:02:49,484 [INFO   ] Installing <Dataset path=/mnt/datasets/datalad/test-crawled2/all-nonrecursive/crcns> from http://datasets.datalad.org/crcns/.git (install.py:378)
1 installed item is available at
<Dataset path=/mnt/datasets/datalad/test-crawled2/all-nonrecursive/crcns>
glalteva has logged on pts/1 from galteva.

commenting out debug stmt didn't help to bring that portion of the bug back .... I am really not sure WTF has happened and it seems that now original install command works... I guess it was the tired mind + stars. I will close for now

[yarikoptic]

not quite sure yet of why the issue comes about or not, but we ran into it again, and it seems at least partially (infinite recursion while installing a subdataset into not yet initialized directory -- PR to fix is coming)...

[bpoldrack]

If you don't mind waiting a few days, I'd propose to not care for it, since I'll rework install anyway. There is all the functionality of add, get, create to remove, so fixing install currently is much more of an effort.

[yarikoptic]

On Thu, 25 Aug 2016, Benjamin Poldrack wrote:

If you don't mind waiting a few days, I'd propose to not care for it,
since I'll rework install anyway. There is all the functionality of add,
get, create to remove, so fixing install currently is much more of an
effort.

As I said, I will not stop anyone fixing install to make it better ;)
but meanwhile I can't promise not fixing something install related (or
in install) ;)

[bpoldrack]

Fix is deprecated with #613, but new implementation doesn't do what the fix was avoiding. So, until someone reproduces this, I'll mark this for closing.