cleaning up add, progress bars and status #6224
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change in behaviour: Previously,
dvc add
would recursively find all files and then expand upon it with glob.This changes to first glob, and then recursively find all files, which matches with what the shell would do.
Fixes progress bar for globbed targets (previously the no. for them was not accounted for).
Reduces progress bars to 2. I have merged targets collection and stage creation into the same generator pipeline and hence replaced by a single progress bar. The other progress bar is during
Adding
.Also adds spinners while collecting cache and checking for graph correctness.
Simplifies large dir recursive add warning
i.e. removes the hint to remove the generated file. As I think if a user makes a mistake, they will cancel the operation in the middle, so no need to remove the generated file.
Makes "recursive large directory add warning" more responsive. It'll print the message as soon as it crosses the threshold of 100, whereas before it'd have traverse through a (potentially) large directory before warning the user.
Reorganises codebase to make it clear what
add
really does.tqdm
progress bars and messages might get interleaved which will be fixed in ui: Prevent tqdm from breaking on output #6280.I am not sure why tqdm progress bar becomes empty at the end and still looks like it's iterating, but it's not.