Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

annexrepo: Tell caller that annex is scanning for unlocked files #4316

Merged
merged 5 commits into from Mar 24, 2020

Conversation

kyleam
Copy link
Collaborator

@kyleam kyleam commented Mar 17, 2020

@yarikoptic mentioned that git annex init could take some time on the "scanning for unlocked files" phase. This series teaches AnnexRepo._run_annex_command to take a custom protocol and switches AnnexRepo._init over to using a WitlessProtocol that logs this at the info level.


Example output:

$ datalad install ///openneuro                  
[INFO   ] Scanning for unlocked files (this may take some time)                         
install(ok): /tmp/dl-vwEPXnr/openneuro (dataset)   

WitlessProtocol calls getpreferredencoding() twice in
_prepare_result().  Let's instead store the value as an attribute
because, as is already the case with the GitProgress protocol, it's
likely that child classes that work with the output will need this.
@kyleam kyleam changed the title annexrepo: Tell caller that annex scanning for unlocked files annexrepo: Tell caller that annex is scanning for unlocked files Mar 17, 2020
@yarikoptic
Copy link
Member

thank you @kyleam - from a quick look seems to be LGTM. I've not tried yet though

@kyleam
Copy link
Collaborator Author

kyleam commented Mar 18, 2020

My added test fails on Travis.

@kyleam kyleam added the WIP work in progress label Mar 18, 2020
@codecov
Copy link

codecov bot commented Mar 18, 2020

Codecov Report

Merging #4316 into master will decrease coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4316      +/-   ##
==========================================
- Coverage   88.83%   88.81%   -0.02%     
==========================================
  Files         285      285              
  Lines       37457    37485      +28     
==========================================
+ Hits        33275    33294      +19     
- Misses       4182     4191       +9     
Impacted Files Coverage Δ
datalad/cmd.py 89.92% <100.00%> (+0.01%) ⬆️
datalad/support/annexrepo.py 86.21% <100.00%> (+0.16%) ⬆️
datalad/support/gitrepo.py 90.76% <100.00%> (-0.01%) ⬇️
datalad/support/tests/test_annexrepo.py 94.92% <100.00%> (-0.35%) ⬇️
datalad/interface/tests/test_unlock.py 95.65% <0.00%> (-2.18%) ⬇️
datalad/core/local/tests/test_save.py 96.25% <0.00%> (-0.50%) ⬇️
datalad/core/distributed/clone.py 91.86% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 64a727c...47386d4. Read the comment docs.

@kyleam
Copy link
Collaborator Author

kyleam commented Mar 18, 2020

My added test fails on Travis.

Older git-annex versions were a bit more selective about showing that message. I've changed the test case so that the message is triggered there too.

@kyleam kyleam removed the WIP work in progress label Mar 18, 2020
_run_annex_command(..., runner="gitwitless") is incompatible with
callables for log_stdout and log_stderr, as mentioned in the to-do
comment from af082fa (RF: make possible to use GitWitlessRunner in
_run_annex_command and use for enable_remote, 2020-02-26).  Raise a
ValueError in this case.
As of af082fa (2020-02-26), _run_annex_command() callers can
specify that GitWitlessRunner should be used instead of Runner.  In
this case, a protocol class is constructed based on the boolean values
of log_std{out,err}.

However, in order to do any custom handling of the output (as is done
in the next commit), the caller has to be able to specify a custom
protocol.  Add a `protocol` parameter to _run_annex_command() so that
they can do so.
Many _run_annex_command() calls have been switched to using
runner="gitwitless".  The call in _init() wasn't a candidate because
it uses a callable for log_stderr, but now we can achieve the same
thing by specifying a custom WitlessProtocol to _run_annex_command().
When init'ing v6+ repos, git-annex scans for unlocked files.  This has
the potential to take some time, so let the user know what's
happening.

Suggested-by: Yaroslav Halchenko <debian@onerussian.com>
@kyleam kyleam added the merge-if-ok OP considers this work done, and requests review/merge label Mar 19, 2020
@kyleam kyleam merged commit 9b30ac2 into datalad:master Mar 24, 2020
@kyleam kyleam deleted the init-relay-annex-unlock branch March 24, 2020 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merge-if-ok OP considers this work done, and requests review/merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants