Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cleaning up add, progress bars and status #6224

Merged
merged 3 commits into from
Jul 5, 2021

Conversation

skshetry
Copy link
Member

@skshetry skshetry commented Jun 25, 2021

  • Change in behaviour: Previously, dvc add would recursively find all files and then expand upon it with glob.
    This changes to first glob, and then recursively find all files, which matches with what the shell would do.

  • Fixes progress bar for globbed targets (previously the no. for them was not accounted for).

  • Reduces progress bars to 2. I have merged targets collection and stage creation into the same generator pipeline and hence replaced by a single progress bar. The other progress bar is during Adding.

  • Also adds spinners while collecting cache and checking for graph correctness.

  • Simplifies large dir recursive add warning
    i.e. removes the hint to remove the generated file. As I think if a user makes a mistake, they will cancel the operation in the middle, so no need to remove the generated file.

  • Makes "recursive large directory add warning" more responsive. It'll print the message as soon as it crosses the threshold of 100, whereas before it'd have traverse through a (potentially) large directory before warning the user.

  • Reorganises codebase to make it clear what add really does.

asciicast

  • Note that the tqdm progress bars and messages might get interleaved which will be fixed in ui: Prevent tqdm from breaking on output #6280.
  • I am not sure why tqdm progress bar becomes empty at the end and still looks like it's iterating, but it's not.

@skshetry skshetry marked this pull request as ready for review June 26, 2021 07:35
@skshetry skshetry requested a review from a team as a code owner June 26, 2021 07:35
@skshetry skshetry requested a review from efiop June 26, 2021 07:35
@skshetry skshetry marked this pull request as draft June 26, 2021 07:35
@skshetry skshetry force-pushed the cleanup-add branch 2 times, most recently from e6f17a0 to 0b03684 Compare July 2, 2021 10:08
@skshetry skshetry self-assigned this Jul 5, 2021
@skshetry skshetry added enhancement Enhances DVC refactoring Factoring and re-factoring ui user interface / interaction labels Jul 5, 2021
@skshetry skshetry added this to In progress in DVC 29 June - 12 July 2021 via automation Jul 5, 2021
@skshetry skshetry moved this from In progress to Review in progress in DVC 29 June - 12 July 2021 Jul 5, 2021
@skshetry skshetry marked this pull request as ready for review July 5, 2021 07:28
@skshetry skshetry changed the title [WIP] cleaning up add, progress bars and status cleaning up add, progress bars and status Jul 5, 2021
@efiop efiop merged commit beb6277 into iterative:master Jul 5, 2021
DVC 29 June - 12 July 2021 automation moved this from Review in progress to Done Jul 5, 2021
@skshetry skshetry deleted the cleanup-add branch July 6, 2021 02:16
@jorgeorpinel
Copy link
Contributor

Very nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhances DVC refactoring Factoring and re-factoring ui user interface / interaction
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

None yet

3 participants