RF: Implement new install/get API #613

bpoldrack · 2016-07-07T10:13:36Z

No actual description. Just referencing things ...

Closes #553
Closes #834
Closes #724
Closes #693
Closes #787

implicit install via get
check what might be untested
should run get if dataset itself was alrady installed ((#613) consecutive install -g doesn't bother getting data #862)
install within not installed subdataset RF: Implement new install/get API #613 (comment)
get -r should just not bother calling get of non Annex repo (super)datasets (#613) get -r should just not bother calling get of non Annex repo (super)datasets #864
Closes can't install dataset referencing using /// datalad RI into existing dataset` #593
hopefully finally Closes install must not fail if installing the same thing again #871 (re-installation into the same dir)

(cherry picked from commit f427b87)

(cherry picked from commit 092db7b)

coveralls · 2016-07-07T10:22:04Z

Coverage increased (+0.04%) to 85.51% when pulling b196cc8 on bpoldrack:rf-get into 797b455 on datalad:master.

codecov-io · 2016-07-07T10:24:05Z

Codecov Report

Merging #613 into master will increase coverage by 0.19%.
The diff coverage is 92.26%.

@@            Coverage Diff             @@
##           master     #613      +/-   ##
==========================================
+ Coverage   87.48%   87.68%   +0.19%     
==========================================
  Files         216      219       +3     
  Lines       19912    20369     +457     
==========================================
+ Hits        17421    17861     +440     
- Misses       2491     2508      +17

Impacted Files	Coverage Δ
datalad/interface/__init__.py	`100% <ø> (ø)`	⬆️
datalad/distribution/add.py	`84.82% <ø> (ø)`	⬆️
datalad/tests/utils.py	`90.07% <ø> (+0.15%)`	⬆️
datalad/distribution/tests/test_dataset.py	`100% <100%> (ø)`	⬆️
datalad/interface/save.py	`96.59% <100%> (+1.13%)`	⬆️
datalad/crawler/nodes/annex.py	`84.39% <100%> (ø)`	⬆️
datalad/distribution/tests/test_get.py	`100% <100%> (ø)`
datalad/distribution/create_test_dataset.py	`84.7% <100%> (-0.18%)`	⬇️
datalad/tests/test_dochelpers.py	`100% <100%> (ø)`	⬆️
datalad/auto.py	`78.46% <100%> (ø)`	⬆️
... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d5d30e0...310bc1e. Read the comment docs.

yarikoptic · 2016-07-14T23:12:39Z

how is it going? I would be excited to try new install and get combination ;)

yarikoptic · 2016-07-18T13:55:48Z

datalad/distribution/get.py

+        annex_get_opts=annex_get_opts)
+
+    @staticmethod
+    @datasetmethod(name='add')


name='get'?

…tasets

coveralls · 2016-09-06T13:15:25Z

Coverage decreased (-2.2%) to 86.674% when pulling 465b985 on bpoldrack:rf-get into 5ff2165 on datalad:master.

coveralls · 2016-09-06T13:15:25Z

Coverage decreased (-2.2%) to 86.674% when pulling 465b985 on bpoldrack:rf-get into 5ff2165 on datalad:master.

coveralls · 2016-09-06T13:15:25Z

Coverage decreased (-2.2%) to 86.674% when pulling 465b985 on bpoldrack:rf-get into 5ff2165 on datalad:master.

mih · 2016-09-06T13:44:05Z

datalad/distribution/get.py

+                                                recursion_limit=recursion_limit)
+            if p_ds is None:
+                raise FileNotInRepositoryError(
+                    msg="{0} not in dataset.".format(p))


Or just a warning that this will be ignored?

May be. Same for not existing files?

mih · 2016-09-06T13:53:43Z

Observation:

% ll -l
total 16K
drwxr-xr-x 3 mih mih 4,0K Sep  1 19:37 deep/
lrwxrwxrwx 1 mih mih  108 Sep  1 19:36 one -> .git/annex/objects/2W/kW/MD5E-s0--d41d8cd98f00b204e9800998ecf8427e/MD5E-s0--d41d8cd98f00b204e9800998ecf8427e
lrwxrwxrwx 1 mih mih  108 Sep  1 19:36 three -> .git/annex/objects/2W/kW/MD5E-s0--d41d8cd98f00b204e9800998ecf8427e/MD5E-s0--d41d8cd98f00b204e9800998ecf8427e
lrwxrwxrwx 1 mih mih  108 Sep  1 19:36 two -> .git/annex/objects/2W/kW/MD5E-s0--d41d8cd98f00b204e9800998ecf8427e/MD5E-s0--d41d8cd98f00b204e9800998ecf8427e
% datalad get one two three
one ... failed.
two ... failed.
three ... failed.

All of the above symlinks already point to existing content.

mih · 2016-09-06T13:55:43Z

Somewhat overwhelmed by the verbosity on standard talkative level:

% datalad get deep/some
2016-09-06 15:54:26,226 [ERROR  ] FileNotInRepositoryError: command ''.
| /tmp/some/deep/some belongs to subdataset <Dataset path=/tmp/some/deep>. To get its content use option `recursive` or call get on the subdataset.
| [Errno None] : /tmp/some/deep/some belongs to subdataset <Dataset path=/tmp/some/deep>. To get its content use option `recursive` or call get on the subdataset.: '' [get.py:__call__:133] (FileNotInRepositoryError) (main.py:259)

bpoldrack · 2016-09-06T13:57:34Z

Re observation on identical content: Yes. Fixed already (partially), but not pushed yet. Result reporting is a topic for hangout. I have questions ... ;-)

…sn't

…ng files.

yarikoptic · 2016-09-08T02:01:21Z

datalad/distribution/get.py

+                resolved_datasets.get(p_ds.path, []) + [p]
+
+            # TODO: Change behaviour of Dataset: Make subdatasets singletons to
+            # always get the same object referencing a certain subdataset.


I would worry about that later whenever we actually run into the problem... for now I think the most important is to get get and install working correctly

Actually, "the problem" is that it currently leads to a less clean and straight implementation of API. But I agree - getting them working correctly again is the more important thing right now.

…with submodules

yarikoptic · 2016-09-20T12:22:28Z

yes

edit: we discussed that before, agreed that it is useful, and we had it implemented (and I thought we had a test for that). So imho it should stay implemented

edit2: later we might want to make it up to configuration variable to do such installations in general or not, and ask user upon initial invocation

bpoldrack · 2016-09-20T12:31:52Z

Well, if you say so, I will add it.

ENH: more concise result renderer for install

…rchy was not an annex. (Closes datalad#864, datalad#862)

and tune it a little bit

yarikoptic · 2016-09-20T13:31:14Z

datalad/distribution/get.py

@@ -262,6 +270,9 @@ def __call__(
                        relpath(opj(ds_path, local_results[i]['file']), ds.path)

            global_results.extend(local_results)
+
+        if not found_an_annex:
+            lgr.warning("Found no annex. Could not perform any get operation.")


I am ok with such approach instead of mine... just fix up the test... I forgot also to improve that test by making an annex subdataset and testing actual recursive call on super pure-git dataset... we really need those tests

just cherry pick that commit ENHing single_or_plural please.

ok -- I will leave you alone for now ;)

yarikoptic · 2016-09-20T13:33:08Z

I think that the merge of this PR is upon us -- basic real life testing identified a few issue, but it is quite good already. so I will mark it as for milestone 0.3 ;-)

(cherry picked from commit 5a6b9a5)

(Closes datalad#593)

yarikoptic · 2016-09-21T13:38:43Z

So we are just 1 quick fix away from the merge! ;) whoohooo -- great job @bpoldrack! It feels like the beast could be usable now! But I would really appreciate if you give a test yourself on our /// and especially those datasets which require authentication (crcns, kaggle, etc)

yarikoptic · 2016-09-21T14:02:39Z

datalad/distribution/install.py

+                            # Keep original in debug output:
+                            lgr.debug("Original failure:{0}"
+                                      "{1}".format(linesep, exc_str(e)))
+                            return None


nope -- not None. Remember -- "fulfilling the promise!" ;) and that is why tests failed as well btw

Sure, None! ;-)
I think you misread it. If there is an installed dataset at the target we return it. But if it is something else we return None since we neither installed a dataset nor is it already there.

But there's something else wrong with it. Condition conflicts with URL guessing, since an empty directory is okay, but could still fail in an attempt to guess the correct URL.

Hopefully works now ;-)

yarikoptic · 2016-09-21T14:02:52Z

datalad/distribution/install.py

+                        # (TODO: eventually check for being the one, that this
+                        # is about)
+                        if current_dataset.is_installed():
+                            lgr.info("{0} appears to be installed already.")


forgotten .format()?

yarikoptic · 2016-09-21T14:04:05Z

datalad/distribution/install.py

+                            break
+                        else:
+                            lgr.warning("Target {0} already exists and is not an "
+                                        "installed dataset. Skipped.")


again forgotten format or I am not aware of some magic? (in python3.5 e.g. you could do smth like r"{path} blah" which would use local var path)

You're right. Committed to fast.

BF: Tested wrong condition get was unnecessarily recursivly called from within install

bpoldrack · 2016-09-21T15:27:26Z

Damn it. OSX failing ...

yarikoptic · 2016-09-21T15:41:21Z

datalad/distribution/install.py

+                            # is about)
+                            if current_dataset.is_installed():
+                                lgr.info("{0} appears to be installed already."
+                                         "".format(current_dataset))


ha - "".format -- cute... ;) thought myself about what to do in such cases ;)

mih and others added 3 commits July 7, 2016 12:10

RF: New common parameters

4bf3514

(cherry picked from commit f427b87)

RF: API proposal for "get"

373558e

(cherry picked from commit 092db7b)

BF: List new command in docs and API

b196cc8

mih mentioned this pull request Jul 7, 2016

DONTMERGE: New set of more compat/homogeneous high-level commands #598

Closed

yarikoptic reviewed Jul 18, 2016
View reviewed changes

datalad/distribution/get.py

annex_get_opts=annex_get_opts)

@staticmethod

@datasetmethod(name='add')

Copy link

Member

yarikoptic Jul 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name='get'?

bpoldrack added 5 commits September 1, 2016 05:12

Merge branch 'rf-add' into rf-get

f054510

line out what to do

c4429c9

Merge branch 'rf-add' into rf-get

6d7c765

First implementation of 'get'; Without implicit installation of subda…

bbb22b9

…tasets

Merge remote-tracking branch 'origin/master' into rf-get

465b985

mih reviewed Sep 6, 2016
View reviewed changes

bpoldrack added 5 commits September 6, 2016 18:18

Change output to rely on whatever annex reports

ecd58c4

Workaround to report on not existing files, which annex currently doe…

52dde9e

…sn't

Adapt 'add' to deal with new return value of annex json on not existi…

00dccca

…ng files.

ENH: Add some logging; Skip somehow invalid paths instead of fail

229292f

BF+TST: Minor fixes; Add some tests

9a73077

yarikoptic reviewed Sep 8, 2016
View reviewed changes

bpoldrack added 2 commits September 8, 2016 13:16

BF: quick checks for file_has_content and is_under_annex didn't work …

fb538d5

…with submodules

TST: More tests for 'get'.

ea1129d

bpoldrack and others added 3 commits September 20, 2016 14:55

Merge pull request #33 from yarikoptic/pr-613

e8efc78

ENH: more concise result renderer for install

BF: recursive get wasn't called correctly when something in the hiera…

5f5de36

…rchy was not an annex. (Closes datalad#864, datalad#862)

Merge remote-tracking branch 'myfork/rf-get' into rf-get

46b7a82

and tune it a little bit

yarikoptic reviewed Sep 20, 2016

View reviewed changes

yarikoptic added this to the Release 0.3 milestone Sep 20, 2016

This was referenced Sep 20, 2016

create-publication-target-sshwebserver does not reuse connection #826

Closed

spit out more informative/meaningful msg whenever some file is not actually accessible bids-standard/legacy-validator#197

Closed

yarikoptic and others added 7 commits September 21, 2016 09:45

ENH: include_count for single_or_plural

1b5e56b

(cherry picked from commit 5a6b9a5)

BF: get_containing_subdataset didn't correctly work recursively

4c8bb0e

TST: Test get more intense on not-all-annex-hierarchies

1a8c5e0

RF: make implicit installation a helper function

f51743a

Shorten description line

aa78046

ENH:Install within not yet installed subdataset

e7ea6d5

BF: _install_from_flexible_source failed to guess /.git ending

001ca96

(Closes datalad#593)

bpoldrack mentioned this pull request Sep 21, 2016

install asks for password twice -- controlpath config is setup but controlpath is not established #865

Closed

Changed behaviour on existing target (Closes datalad#871)

759c684

yarikoptic reviewed Sep 21, 2016

View reviewed changes

yarikoptic requested changes Sep 21, 2016

View reviewed changes

BF: Move exception handling

3baa1ee

BF: Tested wrong condition get was unnecessarily recursivly called from within install

yarikoptic reviewed Sep 21, 2016

View reviewed changes

yarikoptic mentioned this pull request Sep 21, 2016

provide custom abspath -- builtin resolves symlinks (sometimes) #878

Closed

BF: do not use abspath (see datalad#878), join getpwd and the path

310bc1e

yarikoptic merged commit 8a1b234 into datalad:master Sep 21, 2016

yarikoptic mentioned this pull request Sep 21, 2016

(#613) consecutive install -g doesn't bother getting data #862

Closed

yarikoptic deleted the rf-get branch October 17, 2016 12:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RF: Implement new install/get API #613

RF: Implement new install/get API #613

bpoldrack commented Jul 7, 2016 •

edited by yarikoptic

Loading

coveralls commented Jul 7, 2016 •

edited

Loading

codecov-io commented Jul 7, 2016 •

edited by codecov bot

Loading

yarikoptic commented Jul 14, 2016

yarikoptic Jul 18, 2016

coveralls commented Sep 6, 2016 •

edited

Loading

coveralls commented Sep 6, 2016

coveralls commented Sep 6, 2016

mih Sep 6, 2016

bpoldrack Sep 6, 2016

mih commented Sep 6, 2016 •

edited

Loading

mih commented Sep 6, 2016

bpoldrack commented Sep 6, 2016

yarikoptic Sep 8, 2016

bpoldrack Sep 8, 2016

yarikoptic commented Sep 20, 2016 •

edited

Loading

bpoldrack commented Sep 20, 2016

yarikoptic Sep 20, 2016

yarikoptic Sep 20, 2016

yarikoptic commented Sep 20, 2016

yarikoptic commented Sep 21, 2016

yarikoptic Sep 21, 2016

bpoldrack Sep 21, 2016

bpoldrack Sep 21, 2016

yarikoptic Sep 21, 2016

bpoldrack Sep 21, 2016

yarikoptic Sep 21, 2016

bpoldrack Sep 21, 2016

bpoldrack Sep 21, 2016

bpoldrack commented Sep 21, 2016

yarikoptic Sep 21, 2016

RF: Implement new install/get API #613

RF: Implement new install/get API #613

Conversation

bpoldrack commented Jul 7, 2016 • edited by yarikoptic Loading

coveralls commented Jul 7, 2016 • edited Loading

codecov-io commented Jul 7, 2016 • edited by codecov bot Loading

Codecov Report

yarikoptic commented Jul 14, 2016

Choose a reason for hiding this comment

coveralls commented Sep 6, 2016 • edited Loading

coveralls commented Sep 6, 2016

coveralls commented Sep 6, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mih commented Sep 6, 2016 • edited Loading

mih commented Sep 6, 2016

bpoldrack commented Sep 6, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yarikoptic commented Sep 20, 2016 • edited Loading

bpoldrack commented Sep 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yarikoptic commented Sep 20, 2016

yarikoptic commented Sep 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpoldrack commented Sep 21, 2016

Choose a reason for hiding this comment

bpoldrack commented Jul 7, 2016 •

edited by yarikoptic

Loading

coveralls commented Jul 7, 2016 •

edited

Loading

codecov-io commented Jul 7, 2016 •

edited by codecov bot

Loading

coveralls commented Sep 6, 2016 •

edited

Loading

mih commented Sep 6, 2016 •

edited

Loading

yarikoptic commented Sep 20, 2016 •

edited

Loading