Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gc: Completely silent while removing files from remote. #8721

Open
daavoo opened this issue Dec 21, 2022 · 1 comment
Open

gc: Completely silent while removing files from remote. #8721

daavoo opened this issue Dec 21, 2022 · 1 comment
Labels
A: gc Related go garbage collection bug Did we break something? p2-medium Medium priority, should be done, but less important ui user interface / interaction

Comments

@daavoo
Copy link
Contributor

daavoo commented Dec 21, 2022

When removing files from remote storage, DVC stops displaying any updates (no progress bar or log messages).

As described in #5961 (comment) , the actual removal of files can take a while and we just don't inform the user of what's happening.

See the attached video where the silence starts.

gc.mp4

The next debug log message comes after completion, which for the linked reproduction (only 100 files) means >2 minutes without displaying anything (the silent time grows linearly with the number of files).


The issue comes from https://github.com/iterative/dvc-data/blob/main/src/dvc_data/hashfile/gc.py#L28 but I preferred to track it here.

@daavoo daavoo added bug Did we break something? ui user interface / interaction A: gc Related go garbage collection labels Dec 21, 2022
@daavoo
Copy link
Contributor Author

daavoo commented Jan 12, 2023

The proper fix would be to add callbacks to the base fsspec rm and to downstream implementations

@dberenbaum dberenbaum added the p2-medium Medium priority, should be done, but less important label Aug 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: gc Related go garbage collection bug Did we break something? p2-medium Medium priority, should be done, but less important ui user interface / interaction
Projects
None yet
Development

No branches or pull requests

2 participants