Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: enable recursive glob (**) for run #6262

Merged
merged 3 commits into from
Dec 1, 2021
Merged

Conversation

kimsin98
Copy link
Contributor

Closes #6241

Copy link
Member

@adswa adswa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thanks a lot! I just quickly glanced at the test failures for now, I believe they stem from a misspelled key in the examples - left a quick suggestion for that.

datalad/core/local/run.py Outdated Show resolved Hide resolved
@adswa adswa added the semver-patch Increment the patch version when merged label Nov 30, 2021
@adswa
Copy link
Member

adswa commented Nov 30, 2021

added semver-patch for the semver test, please change or say something if someone disagrees :)

Co-authored-by: Adina Wagner <adina.wagner@t-online.de>
@codecov
Copy link

codecov bot commented Nov 30, 2021

Codecov Report

Merging #6262 (00eee07) into master (b7ed22e) will decrease coverage by 3.40%.
The diff coverage is 100.00%.

❗ Current head 00eee07 differs from pull request most recent head 22bf7fa. Consider uploading reports for the commit 22bf7fa to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6262      +/-   ##
==========================================
- Coverage   89.62%   86.21%   -3.41%     
==========================================
  Files         323      323              
  Lines       41984    41979       -5     
==========================================
- Hits        37629    36194    -1435     
- Misses       4355     5785    +1430     
Impacted Files Coverage Δ
datalad/core/local/run.py 97.78% <ø> (ø)
datalad/support/globbedpaths.py 100.00% <100.00%> (ø)
datalad/support/tests/test_globbedpaths.py 95.06% <100.00%> (-4.94%) ⬇️
datalad/version.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/plugin/wtf.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/plugin/addurls.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/interface/clean.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/interface/rerun.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/plugin/no_annex.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/interface/unlock.py 0.00% <0.00%> (-100.00%) ⬇️
... and 166 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7ed22e...22bf7fa. Read the comment docs.

Copy link
Member

@adswa adswa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works nicely for me! Thanks a lot! I will wait a bit to give others a chance to spot things I may have missed, though :)

@mih mih added semver-minor Increment the minor version when merged and removed semver-patch Increment the patch version when merged labels Nov 30, 2021
Copy link
Member

@mih mih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thx!

@yarikoptic
Copy link
Member

yarikoptic commented Nov 30, 2021

Looks great! My only "fear"s are

  • that it might change reproducibility of prior runs. E.g. if someone used ** just by doubletyping *, which resulted in non-recursive match, now behavior would change, so rerun might find more. edit: but I think it is Ok as long as we clearly document such a change in CHANGELOG for the release. Nothing should stop the progress! (a complimentary -- there could be yet another config option to disable recursive=True)
  • in presence of subdatasets, something like sub/**/*.nii.gz could find even git annex keys themselves within sub/.git/annex/objects . But it seems to be not the case, in that ** I guess does not include dot dirs:
$> python3 -c "from glob import glob; print([x for x in glob('ds*/.git/**/*.nii.gz', recursive=True) if 'annex' in x])"
['ds003027/.git/annex/objects/ZJ/k5/MD5E-s89045275--1625ff12d9c2d4e35f50df32601a7bb2.nii.gz', 'ds003027/.git/annex/objects/ZJ/k5/MD5E-s89045275--1625ff12d9c2d4e35f50df32601a7bb2.nii.gz/MD5E-s89045275--1625ff12d9c2d4e35f50df32601a7bb2.nii.gz']

$> python3 -c "from glob import glob; print([x for x in glob('ds*/**/*.nii.gz', recursive=True) if 'annex' in x])"     
[]

so I guess we should be safe on that end.

@mih
Copy link
Member

mih commented Dec 1, 2021

OK, let's proceed! A changelog will come.

@mih mih merged commit a9e5201 into datalad:master Dec 1, 2021
@adswa
Copy link
Member

adswa commented Dec 1, 2021

Its mentioned in the handbook now as well 👍

@mih
Copy link
Member

mih commented Dec 1, 2021

Great, thx!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver-minor Increment the minor version when merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support recursive wildcards for run output specifications
5 participants