Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large code-reorganization of everything runner-related #6008

Merged
merged 14 commits into from Sep 27, 2021
Merged

Conversation

mih
Copy link
Member

@mih mih commented Sep 24, 2021

Basically, this re-organizes the code to establish datalad.runner as a self-contained subpackage that include all runner-related functionality, and no other functionality. Legacy imports are put in place to minimize likelihood of API breakage.

  • clear separation of runner from non-runner related code (ie. BatchedCommand is a different animal)
  • clearly separated tests
  • clear separation of runner interface, from protocol interface, from their various implementations, and underlying helpers
  • clear indication of necessary helpers from the rest of datalad
  • place CommandError with the runner code, as they are tightly linked
  • adjust appveyor/travis setup to consider the new test location

This is proposed against maint to avoid carrying a large diff for the full 0.16 release cycle. it should cause no breakage.

This is the core exception type used, and it is only raised by the runner
(except for test code).

Reimport to its previous place to avoid breakage.
Not needed elsewhere, but re-import to avoid breakage.
The non-runner execution (BatchedCommand) is put back in its
original place, where it was before this RF.
These (alone) give
```
Name                               Stmts   Miss  Cover   Missing
----------------------------------------------------------------
datalad/runner/__init__.py             4      0   100%
datalad/runner/coreprotocols.py       19      2    89%   53-54
datalad/runner/exception.py           38      6    84%   56, 59-63, 67, 70
datalad/runner/gitrunner.py           72     13    82%   51-55, 68, 87, 94-96, 186, 197-198
datalad/runner/nonasyncrunner.py      96      1    99%   111
datalad/runner/protocol.py            57      3    95%   74, 89-93
datalad/runner/runner.py              36      0   100%
datalad/runner/utils.py                3      0   100%
----------------------------------------------------------------
TOTAL                                325     25    92%
----------------------------------------------------------------------
Ran 21 tests in 9.602s
```
@codecov
Copy link

codecov bot commented Sep 24, 2021

Codecov Report

Merging #6008 (38d1659) into maint (611c371) will decrease coverage by 1.79%.
The diff coverage is 89.10%.

❗ Current head 38d1659 differs from pull request most recent head 0c5b755. Consider uploading reports for the commit 0c5b755 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##            maint    #6008      +/-   ##
==========================================
- Coverage   90.28%   88.48%   -1.80%     
==========================================
  Files         308      312       +4     
  Lines       42103    42131      +28     
==========================================
- Hits        38012    37281     -731     
- Misses       4091     4850     +759     
Impacted Files Coverage Δ
datalad/core/local/run.py 97.78% <ø> (-0.01%) ⬇️
datalad/distributed/create_sibling_gitlab.py 74.83% <ø> (-0.17%) ⬇️
datalad/distributed/export_archive_ora.py 80.30% <ø> (-0.30%) ⬇️
datalad/local/check_dates.py 66.66% <ø> (-29.05%) ⬇️
datalad/local/subdatasets.py 93.96% <ø> (-2.62%) ⬇️
datalad/local/tests/test_wtf.py 81.70% <0.00%> (-8.66%) ⬇️
datalad/core/distributed/clone.py 91.05% <31.57%> (-1.38%) ⬇️
datalad/core/local/resulthooks.py 70.96% <33.33%> (+0.47%) ⬆️
datalad/local/wtf.py 82.05% <33.33%> (-0.37%) ⬇️
datalad/runner/gitrunner.py 90.27% <90.27%> (ø)
... and 89 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 611c371...0c5b755. Read the comment docs.

@mih mih added the semver-patch Increment the patch version when merged label Sep 25, 2021
Use relative imports for runner components using the symbols from
datalad.runner

Split off cmd.py test code into a separate module.
@mih
Copy link
Member Author

mih commented Sep 25, 2021

crawler failure is same old datalad/datalad-crawler#104

@mih mih added the merge-if-ok OP considers this work done, and requests review/merge label Sep 26, 2021
@mih mih added this to To be considered in Release 1.0 via automation Sep 27, 2021
Copy link
Member

@bpoldrack bpoldrack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think I can meaningfully double check the whole diff, but I like the aim/approach.

Copy link
Contributor

@christian-monch christian-monch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good IMO. Nice cleanup of dependencies and better separation of sub-systems

@mih
Copy link
Member Author

mih commented Sep 27, 2021

Thanks for the feedback, will merge now.

@mih mih merged commit d183fd7 into datalad:maint Sep 27, 2021
Release 1.0 automation moved this from To be considered to DONE Sep 27, 2021
@mih mih deleted the rf-runner branch September 27, 2021 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merge-if-ok OP considers this work done, and requests review/merge semver-patch Increment the patch version when merged
Projects
Release 1.0
  
DONE
Development

Successfully merging this pull request may close these issues.

None yet

3 participants