BF+RF+ENH: cocktail of changes while was working on crcns/smaug #746

yarikoptic · 2016-09-01T02:42:42Z

added "private" argument to .commit to add our [DATALAD] prefix which now used in many spots
used create command to initiate dataset
--no-commit to create (if we know that we will commit anyways later)
parametric renames (outside of tarballs and on extracted content) for crcns pipeline
add_to_super="auto" to add to super only if there is a super
fixed a test url to db.hcp url
request non-compressed content so our logic of checking download by size to match header works and (ideally a TODO item to enable back, but may be later)

since that breaks our checks for the target file size... TODO to fix it properly but atm otherwise breaks to download crcns list of datasets

that is why test was failing since it was asking for versioning support from non-versioned bucket it asked before about (tests rule! do not ignore them! ;) ) meanwhile fixed few minor issues to robustify ls

…r tests

…ns pipeline more, use create in initialize_dataset, etc

coveralls · 2016-09-01T03:04:26Z

Coverage decreased (-0.04%) to 86.562% when pulling eb0cb56 on yarikoptic:bf-crcns into 64730e6 on datalad:master.

yarikoptic · 2016-09-01T13:09:06Z

funny failure on py 3.3 only

Traceback (most recent call last):

  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/home/travis/build/datalad/datalad/datalad/tests/test_strings.py", line 25, in test_apply_replacement_rules

    eq_(apply_replacement_rules(['/a/b', ',b,ab'], 'abab'), 'abababab')

AssertionError: 'bbbbbb' != 'abababab'

codecov-io · 2016-09-01T13:44:42Z

Codecov Report

Merging #746 into master will decrease coverage by 0.02%.
The diff coverage is 84.74%.

@@            Coverage Diff             @@
##           master     #746      +/-   ##
==========================================
- Coverage   86.79%   86.76%   -0.03%     
==========================================
  Files         200      200              
  Lines       18124    18184      +60     
==========================================
+ Hits        15731    15778      +47     
- Misses       2393     2406      +13

Impacted Files	Coverage Δ
datalad/crawler/pipelines/tests/test_openfmri.py	`90.45% <ø> (ø)`	⬆️
datalad/downloaders/tests/test_http.py	`81.74% <ø> (ø)`	⬆️
datalad/crawler/pipelines/tests/test_balsa.py	`100% <ø> (ø)`	⬆️
datalad/interface/add_archive_content.py	`90.21% <ø> (ø)`	⬆️
datalad/downloaders/tests/test_s3.py	`67.92% <0%> (-1.31%)`	⬇️
datalad/downloaders/base.py	`74.23% <0%> (-0.33%)`	⬇️
datalad/support/s3.py	`27.09% <0%> (ø)`	⬆️
datalad/interface/ls.py	`52.56% <0%> (-0.42%)`	⬇️
datalad/support/gitrepo.py	`88.28% <100%> (+0.12%)`	⬆️
datalad/downloaders/http.py	`83.73% <100%> (+0.15%)`	⬆️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 82df4b4...4f74450. Read the comment docs.

* origin/master: BF: Handle crazy cfg TYPO: contraint -> constraint ENH: use @_where_reload decorator ENH+RF: remove_section, centralized duplicate docstring for where and reload ENH+TODO: test global gitconfig modifications with mocked HOME RF: swap logic - first dataset, and then git configs RF: move all invocations of git config into one method (+ few typos fixed) TST: For the dataset integration of the config DOC: Small fixes DOC: for ConfigManager BF: Make test deterministic NF: Draft of our own config parser Conflicts: datalad/distribution/create.py - took my version with Michael's added code for .origin check/set

coveralls · 2016-09-01T19:16:56Z

Coverage decreased (-14.4%) to 72.403% when pulling 1d9f555 on yarikoptic:bf-crcns into 9a6ad90 on datalad:master.

* origin/master: BF: Deal with symlinks RF: Do not resolve symlinks to realpath

coveralls · 2016-09-01T22:38:56Z

Coverage decreased (-0.04%) to 86.722% when pulling 96b3c38 on yarikoptic:bf-crcns into 8c42f85 on datalad:master.

coveralls · 2016-09-01T22:38:56Z

Coverage decreased (-0.04%) to 86.722% when pulling 96b3c38 on yarikoptic:bf-crcns into 8c42f85 on datalad:master.

coveralls · 2016-09-01T22:38:56Z

Coverage decreased (-0.04%) to 86.722% when pulling 96b3c38 on yarikoptic:bf-crcns into 8c42f85 on datalad:master.

coveralls · 2016-09-02T01:19:25Z

Coverage decreased (-0.03%) to 86.738% when pulling 005f4a6 on yarikoptic:bf-crcns into 8c42f85 on datalad:master.

yarikoptic · 2016-09-02T01:29:55Z

.travis.yml

@@ -8,6 +8,9 @@ cache:
 matrix:
  include:
  - python: 2.7
+    env:
+    - secure: "k2rHdTBjUU3pUUASqfRr7kHaaSmNKLLAR2f66A0fFSulih4CXxwLrR3g8/HP9m+jMve8mAYEiPSI7dT7cCm3WMA/piyLh2wKCGgzDD9oLjtvPAioR8dgLpzbgjxV/Vq6fwwPMlvbqqa+MmAImnAoSufEmI7zVQHCq11Hd5nd6Es="
+    - secure: "Az7Pzo5pSdAFTTX6XXzE4VUS3wnlIe1vi/+bfHBzDjxstDvZVkPjPzaIs6v/BLod43AYBl1v9dyJR4qnBnaVrUDLB3tC0emLhJ2qnw+8GKHSSImCwIOeZzg9QpXeVQovZUqQVQ3fg3KIWCIzhmJ59EbMQcI4krNDxk4WcXmyVfk="


unfortunately doesn't work yet :-/ but better to have the rest merged since conflicts would pile up @mih ;)

yarikoptic · 2016-09-02T15:32:31Z

uf... merge conflicts again/already... if no objections or further comments -- I will merge "master merged" version in an hour or so -- this one includes a number of fixes to tests etc I need to get in

* origin/master: (29 commits) Evasive maneuver (symlink issue, unrelated to this PR) BF: Let `save -a` commit modified submodules too (closes: dataladgh-636) RF: Redesign Dataset.id -- now also for non-annex ones BF: Correctly recurse into subdatasets beneath a given path RF: Consider the dict API as primary for ConfigManager BF: Trivial fixes ENH: Provide return value and user output for add command RF: Use require_dataset BF: Minor fixes after merging master BF: Minor fixes TST: more testing of add API BF+TST: Fail on 'git addurl'; testing 'addurl' BF: get_git_url didn't work with URL instances BF: dataset resolution was incomplete BF: Make get_containing_subdatasetproperly deal with explicit paths TST+BF: Fix (most) tests + minor bugfixes ENH: First implementation of new API for add RF: Make get_containing_subdataset a emthod of Dataset Start actual implementation of add ENH: recursive and recursion_limit for get_conatining_subdataset TST: Shaping tests for add ... Conflicts: datalad/distribution/create.py -- resolved by looking at original merged PRs and adopting their changes into my changes datalad/support/annexrepo.py -- minor conflicts here -- just kept both introduced kwargs datalad/support/gitrepo.py

coveralls · 2016-09-02T16:07:27Z

Coverage decreased (-0.03%) to 86.769% when pulling 4f74450 on yarikoptic:bf-crcns into 82df4b4 on datalad:master.

coveralls · 2016-09-02T16:07:27Z

Coverage decreased (-0.03%) to 86.769% when pulling 4f74450 on yarikoptic:bf-crcns into 82df4b4 on datalad:master.

coveralls · 2016-09-02T16:07:27Z

Coverage decreased (-0.03%) to 86.769% when pulling 4f74450 on yarikoptic:bf-crcns into 82df4b4 on datalad:master.

yarikoptic added 6 commits August 31, 2016 10:47

BF(workaround): request content to not be decompressed on the fly

329d1f9

since that breaks our checks for the target file size... TODO to fix it properly but atm otherwise breaks to download crcns list of datasets

BF: do not reuse previous bucket if it is a different one!

9a68e4e

that is why test was failing since it was asking for versioning support from non-versioned bucket it asked before about (tests rule! do not ignore them! ;) ) meanwhile fixed few minor issues to robustify ls

RF: replacing dsiappeared file from db.hcp with another one

baaa834

BF(TST): reset providers so prev vcr'ed bucket doesn't leak into othe…

39f81e2

…r tests

ENH+BF: crcns inspired cocktail - [DATALAD] commit prefixes, flex crc…

a4f50ff

…ns pipeline more, use create in initialize_dataset, etc

BF: oops -- there should be no skip_no_changes=False in "production"

eb0cb56

BF: use ordered dict while converting replacement rules

0b7886b

yarikoptic added 2 commits September 1, 2016 16:53

Merge remote-tracking branch 'origin/master' into bf-crcns

a36ff50

* origin/master: BF: Deal with symlinks RF: Do not resolve symlinks to realpath

BF(TST): for now carrying .datalad/config in every dataset

96b3c38

yarikoptic added 2 commits September 1, 2016 20:40

ENH: allow credentials to be provided via env vars

d644c75

Let's try to enable some testing on s3

005f4a6

yarikoptic reviewed Sep 2, 2016
View reviewed changes

yarikoptic mentioned this pull request Sep 2, 2016

RF: Implement new API (add) #612

Merged

yarikoptic merged commit cf58158 into datalad:master Sep 2, 2016

yarikoptic deleted the bf-crcns branch September 17, 2016 15:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BF+RF+ENH: cocktail of changes while was working on crcns/smaug #746

BF+RF+ENH: cocktail of changes while was working on crcns/smaug #746

yarikoptic commented Sep 1, 2016

coveralls commented Sep 1, 2016 •

edited

Loading

yarikoptic commented Sep 1, 2016

codecov-io commented Sep 1, 2016 •

edited by codecov bot

Loading

coveralls commented Sep 1, 2016

coveralls commented Sep 1, 2016 •

edited

Loading

coveralls commented Sep 1, 2016 •

edited

Loading

coveralls commented Sep 1, 2016

coveralls commented Sep 2, 2016 •

edited

Loading

yarikoptic Sep 2, 2016

yarikoptic commented Sep 2, 2016

coveralls commented Sep 2, 2016 •

edited

Loading

coveralls commented Sep 2, 2016

coveralls commented Sep 2, 2016

BF+RF+ENH: cocktail of changes while was working on crcns/smaug #746

BF+RF+ENH: cocktail of changes while was working on crcns/smaug #746

Conversation

yarikoptic commented Sep 1, 2016

coveralls commented Sep 1, 2016 • edited Loading

yarikoptic commented Sep 1, 2016

codecov-io commented Sep 1, 2016 • edited by codecov bot Loading

Codecov Report

coveralls commented Sep 1, 2016

coveralls commented Sep 1, 2016 • edited Loading

coveralls commented Sep 1, 2016 • edited Loading

coveralls commented Sep 1, 2016

coveralls commented Sep 2, 2016 • edited Loading

yarikoptic Sep 2, 2016

Choose a reason for hiding this comment

yarikoptic commented Sep 2, 2016

coveralls commented Sep 2, 2016 • edited Loading

coveralls commented Sep 2, 2016

coveralls commented Sep 2, 2016

coveralls commented Sep 1, 2016 •

edited

Loading

codecov-io commented Sep 1, 2016 •

edited by codecov bot

Loading

coveralls commented Sep 1, 2016 •

edited

Loading

coveralls commented Sep 1, 2016 •

edited

Loading

coveralls commented Sep 2, 2016 •

edited

Loading

coveralls commented Sep 2, 2016 •

edited

Loading