ENH: Enable progress bars for unlock, fixes #6691 #6704
Conversation
Codecov Report
@@ Coverage Diff @@
## master #6704 +/- ##
==========================================
+ Coverage 88.81% 90.40% +1.59%
==========================================
Files 353 353
Lines 45733 45742 +9
==========================================
+ Hits 40617 41353 +736
+ Misses 5116 4389 -727
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
LGTM, thx!
Changes like this need to be done all over the code base.
If by 'total counter' you refer to the larger number on the right, it us not a total but a speed estimate. I agree that it is generally pretty useless and inaccurate, but technically not wrong.
oh sorry 🤦 |
An unlocking operation can take a considerable amount of time, but doesn't provide feedback at all (see datalad#6691). By switching from _call_annex_records(), which processes the full bunch of files at once, to _call_annex_records_items(), which yields as soon as results are available, we can construct a progress bar with a file counter around the unlock operation. The progress reporting is slightly complicated by the fact that git annex' --json records for each file are output before a final 'recording state in git' operations, which can, if a large number of files was unlocked, add considerably to the execution time of the command after all files have been unlocked already and the progress bar would be at 100%. Thus, this change adds an additional progress update when all files are processed to report on the final recording step.
|
Code Climate has analyzed commit 01d2619 and detected 0 issues on this pull request. View more on Code Climate. |
Posting this error here in order to help find this PR later. But if this has anything to do with this PR, it would be an indication of a runner issue. If there is such an issue, the runner would need to be fixed. Because AFAICS the usage pattern in this PR is pretty normal. Maybe @christian-monch can comment. |
Sorry, but I cannot reproduce the pause locally. Let's try to figure out why I do see a strange behavior in |
|
Thanks for looking into this! I don't see anything unusual in save - its rapid. Might suggest something the takes place during unlock for me happens during your save? |
This looks like it's caused by a change in Git behavior. Not sure which test produced the error, but I ran into the same error message on our cluster when cloning a dataset owned by another user (as an input subdataset). Note that the failing test on MacOS (brew) had git 2.36.1, while the successful one (snapshot) had 2.35.1 In addition to the issues pointed by @adswa , the error appears in a few others. To tie things together: it was succinctly summarized by @yarikoptic in this issue comment and #6708 The summary (for my presentation of the problem, not sure if appplicable here) is that Git >= 2.35.2 will refuse to operate on |
|
FWIW, looks nice and clean to me, so let's proceed |
An unlocking operation can take a considerable amount of
time, but doesn't provide feedback at all (see #6691).
By switching from _call_annex_records(), which processes
the full bunch of files at once, to _call_annex_records_items(),
which yields as soon as results are available, we can
construct a progress bar with a file counter around the
unlock operation.
The progress reporting is slightly complicated by the
fact that git annex' --json records for each file are
output before a final 'recording state in git' operations,
which can, if a large number of files was unlocked, add
considerably to the execution time of the command after
all files have been unlocked already and the progress bar
would be at 100%. Thus, this change adds an additional
progress update when all files are processed to report
on the final recording step.
Nevertheless, the progress bar seems a bit moody - it seems that the
_call_annex_records_items()loop doesn't yield all files' results before the "recording state in git" stage, leading to a bit of a pause in the middle of the progress bar. Maybe @christian-monch knows what might cause this?Another TODO/Weirdness is a for some reason completely made up file counter (you can see it in the reported file total in the Gif as well) that I wasn't able to figure out. Every run reports a different wrong total number of files, when the total being declared in the setup of the progressbar (
nfiles) is actually correct.Reproducer:
I think I have seen similar flaky reporting in other commands (
push?) so I suspect its something more general with the progress bar setup.Changelog
💫 Enhancements and new features
datalad unlockgained a progress bar.