Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`distribution.subdatasets` -> `local.subdatasets` #3429

Merged
merged 24 commits into from May 21, 2019

Conversation

@mih
Copy link
Member

commented May 17, 2019

This PR implements the transition of the subdatasets command to the "local" API section #3192 . It aligns arguments, behavior, and result structure with the rest "core.local" API.

Python API breakage is caused by the necessary introduction of positional PATH arguments (like the other commands have) in conjunction with the requirement for uniform argument order.

  • multi path matching with --contains
  • supports path-constrained queries like status/diff/save
  • when invoked on the cmdline with no args, reports only subdatasets underneath curdir
  • result content now aligned with status/diff/save, e.g. revision -> gitshasum (but revision is kept as a duplicate for compatibility)
  • no custom parser for subdatasets anymore
  • no custom parser for .gitmodules anymore
  • no GitPy functionality used anymore
  • move to local/subdatasets.py
@yarikoptic
Copy link
Member

left a comment

left suggestion + for supporting submods with '.' in the name

datalad/distribution/subdatasets.py Outdated Show resolved Hide resolved
datalad/distribution/subdatasets.py Outdated Show resolved Hide resolved
@codecov

This comment has been minimized.

Copy link

commented May 18, 2019

Codecov Report

Merging #3429 into master will decrease coverage by 54.96%.
The diff coverage is 22.44%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #3429       +/-   ##
===========================================
- Coverage    91.3%   36.33%   -54.97%     
===========================================
  Files         265      263        -2     
  Lines       34488    34460       -28     
===========================================
- Hits        31488    12520    -18968     
- Misses       3000    21940    +18940
Impacted Files Coverage Δ
datalad/distribution/tests/test_subdataset.py 0% <0%> (-100%) ⬇️
datalad/distribution/subdatasets.py 26.82% <22.91%> (-67.91%) ⬇️
datalad/customremotes/tests/__init__.py 0% <0%> (-100%) ⬇️
datalad/metadata/extractors/__init__.py 0% <0%> (-100%) ⬇️
datalad/distribution/tests/test_clone.py 0% <0%> (-100%) ⬇️
datalad/interface/tests/test_annotate_paths.py 0% <0%> (-100%) ⬇️
datalad/cmdline/tests/__init__.py 0% <0%> (-100%) ⬇️
datalad/metadata/extractors/tests/__init__.py 0% <0%> (-100%) ⬇️
datalad/interface/tests/test_download_url.py 0% <0%> (-100%) ⬇️
datalad/plugin/tests/test_export_archive.py 0% <0%> (-100%) ⬇️
... and 193 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b59bcb0...2f7f554. Read the comment docs.

@codecov

This comment has been minimized.

Copy link

commented May 18, 2019

Codecov Report

Merging #3429 into master will increase coverage by 0.08%.
The diff coverage is 94.37%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3429      +/-   ##
==========================================
+ Coverage    91.2%   91.28%   +0.08%     
==========================================
  Files         265      267       +2     
  Lines       34489    34486       -3     
==========================================
+ Hits        31457    31482      +25     
+ Misses       3032     3004      -28
Impacted Files Coverage Δ
datalad/interface/__init__.py 100% <ø> (ø) ⬆️
datalad/support/gitrepo.py 90.18% <100%> (+0.21%) ⬆️
datalad/distribution/subdatasets.py 100% <100%> (+5.26%) ⬆️
datalad/local/tests/test_subdataset.py 100% <100%> (ø)
datalad/local/__init__.py 100% <100%> (ø)
datalad/plugin/tests/test_addurls.py 100% <100%> (ø) ⬆️
datalad/local/subdatasets.py 93.12% <93.12%> (ø)
datalad/downloaders/http.py 83.73% <0%> (-2.78%) ⬇️
datalad/utils.py 87.28% <0%> (-0.11%) ⬇️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cddb04c...b9a6ad2. Read the comment docs.

@mih mih changed the title [WIP] Some love for `subdatasets` `distribution.subdatasets` -> `local.subdatasets` May 19, 2019

@mih

This comment has been minimized.

Copy link
Member Author

commented May 19, 2019

This is now ready for review.

@mih mih requested review from kyleam and yarikoptic May 19, 2019

@mih mih added this to the Release 0.12.0 milestone May 19, 2019

@mih mih force-pushed the mih:enh-subdatasets branch from 95deb03 to 94908b2 May 19, 2019

mih added some commits May 20, 2019

@mih mih referenced this pull request May 20, 2019
5 of 5 tasks complete
# no constraints given -> query subdatasets under curdir
if path is None and dataset is None:
path = os.curdir
paths = [rev_resolve_path(p, dataset) for p in assure_list(path)] \

This comment has been minimized.

Copy link
@mih

mih May 20, 2019

Author Member

Could be simplified once #3435 is merged, but no requirement to get it done before this PR can be merged.

@mih

This comment has been minimized.

Copy link
Member Author

commented May 20, 2019

Remaining test failure is the already-known neuroimaging extension test run failure, due to a missing config procedure.

mih and others added some commits May 20, 2019

@kyleam
Copy link
Member

left a comment

The command-line subdirectory behavior isn't working as intended. Otherwise looks good to me.

if parser is not None:
# release parser lock manually, auto-cleanup is not reliable in PY3
parser.release()
# Import command class to ease 3rd-party transitions

This comment has been minimized.

Copy link
@kyleam

kyleam May 20, 2019

Member

Should we issue a warning (probably with warnings.warn() rather than logging.warn()) when this module is imported?

This comment has been minimized.

Copy link
@mih

mih May 20, 2019

Author Member

I don't think it is needed. Even in our own code base we did not import it directly.

This comment has been minimized.

Copy link
@kyleam
datalad/local/subdatasets.py Outdated Show resolved Hide resolved
datalad/local/subdatasets.py Show resolved Hide resolved
@yarikoptic

This comment has been minimized.

Copy link
Member

commented May 20, 2019

I did upload fresh -neuroimaging with the fix, and restarted one sample run which was failing -- https://travis-ci.org/datalad/datalad/builds/534685969?utm_source=github_status&utm_medium=notification -- all green now back then a few commits back before ci skips

@kyleam

kyleam approved these changes May 20, 2019

@mih

This comment has been minimized.

Copy link
Member Author

commented May 21, 2019

All green. @yarikoptic didnt update his review, but I think I successfully addressed his requests.

@mih mih merged commit 95d2c0e into datalad:master May 21, 2019

5 checks passed

WIP ready for review
Details
codecov/patch 94.37% of diff hit (target 91.2%)
Details
codecov/project 91.28% (+0.08%) compared to cddb04c
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@mih mih deleted the mih:enh-subdatasets branch May 21, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.