NF: status(eval_subdataset_state={'no'|'commit'|'full'}

[mih]

New mode to fully or partially disable subdataset state evaluation (which involves recursion into all present subdatasets to find potential uncommited changes), without disabling subdataset discovery and
reporting.

This addition allows for using status() unconditionally for (fast(er)) discovery operation.

This also replaces the (undocumented) argument and behavior switch 'ignore_submodules' that wasn't used in any sensible way, because it didn't actually provide sensible switching. It could only distinguish
"clever" operation and "needlessly wasteful" operation.

• final docs

Performance info on the /// dataset:

datalad status --eval-subdataset-state commit -r 744.29s user 601.62s system 102% cpu 21:47.48 total

Looks ridiculous, but a plain git status at the top-level already takes 7min. In contrast this is a recursion across all datasets, with reporting on all datasets. Interestingly, the difference to "full" on the present state of that dataset's working tree (mixture of all kinds of things) is only 2min (~24min total runtime).

So here is an artificial "worst case" example with a deep and clean hierarchy that show maximum performance difference between the modes "commit" and "full":

import datalad.api as dl
ds = dl.rev_create('deep')
path = ds.pathobj  
for i in range(20):       
     path = path / 'down'
     ds.rev_create(path)

%timeit ds.status(eval_subdataset_state='no', result_renderer='disabled')
10 loops, best of 3: 51.8 ms per loop                                                    

%timeit ds.status(eval_subdataset_state='commit', result_renderer='disabled')
10 loops, best of 3: 66.6 ms per loop

%timeit ds.status(eval_subdataset_state='full', result_renderer='disabled')
1 loop, best of 3: 1.07 s per loop

[codecov]

Codecov Report

Merging #3324 into master will decrease coverage by <.01%.
The diff coverage is 93.93%.

@@            Coverage Diff             @@
##           master    #3324      +/-   ##
==========================================
- Coverage   91.15%   91.15%   -0.01%     
==========================================
  Files         263      263              
  Lines       34230    34246      +16     
==========================================
+ Hits        31204    31218      +14     
- Misses       3026     3028       +2

Impacted Files	Coverage Δ
datalad/support/annexrepo.py	87.31% <ø> (ø)	⬆️
datalad/core/local/status.py	97.91% <100%> (ø)	⬆️
datalad/support/tests/test_repo_save.py	100% <100%> (ø)	⬆️
datalad/support/tests/test_fileinfo.py	100% <100%> (ø)	⬆️
datalad/support/gitrepo.py	89.12% <92%> (+0.01%)	⬆️
datalad/downloaders/http.py	83.73% <0%> (-2.78%)	⬇️
datalad/downloaders/tests/test_http.py	92.57% <0%> (+1.48%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5472821...b2bccf7. Read the comment docs.

[kyleam]

I don't have a solid understanding of the underlying code, but the overall idea sounds good to me and the results looked good in the hierarchies I played around with. Pushed a couple fixups (feel free to squash in if you prefer) and left minor comments.

[kyleam]

This seems useful enough to get a short flag (-e?). Also, perhaps setting metavar to something shorter (STATE?) would make the short help a bit more readable. Currently it says

[--eval-subdataset-state EVAL_SUBDATASET_STATE]

[mih]

True, will do.

[mih]

metavar issue is addressed via #3326

[mih]

@kyleam Thx for the feedback. I added a performance note to the top comment.