Skip to content

Conversation

@pmrowla
Copy link
Contributor

@pmrowla pmrowla commented Feb 28, 2023

Thank you for the contribution - we'll try to review it as soon as possible. πŸ™

Closes: #8676
Related: #8787

  • Adds dvc exp clean to cleanup unneeded files from .dvc/tmp/exps. Currently only does dvc-task/celery garbage collection
  • Celery messages are now automatically garbage collected in the background on queue worker exit (when a complete dvc queue start or dvc exp run --run-all batch finishes)

@pmrowla pmrowla added performance improvement over resource / time consuming tasks A: experiments Related to dvc exp labels Feb 28, 2023
@pmrowla pmrowla self-assigned this Feb 28, 2023
@pmrowla pmrowla requested a review from dberenbaum February 28, 2023 09:11
@codecov
Copy link

codecov bot commented Feb 28, 2023

Codecov Report

Patch coverage: 97.43% and project coverage change: -0.01 ⚠️

Comparison is base (286f4fd) 92.97% compared to head (e5edf0c) 92.96%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9090      +/-   ##
==========================================
- Coverage   92.97%   92.96%   -0.01%     
==========================================
  Files         459      461       +2     
  Lines       37113    37150      +37     
  Branches     5369     5370       +1     
==========================================
+ Hits        34504    34535      +31     
- Misses       2082     2085       +3     
- Partials      527      530       +3     
Impacted Files Coverage Ξ”
dvc/commands/experiments/__init__.py 100.00% <ΓΈ> (ΓΈ)
dvc/repo/experiments/queue/celery.py 85.66% <66.66%> (-2.12%) ⬇️
dvc/commands/experiments/clean.py 100.00% <100.00%> (ΓΈ)
dvc/commands/experiments/queue_worker.py 100.00% <100.00%> (ΓΈ)
dvc/repo/experiments/__init__.py 87.23% <100.00%> (+0.13%) ⬆️
dvc/repo/experiments/clean.py 100.00% <100.00%> (ΓΈ)
tests/func/experiments/test_experiments.py 100.00% <100.00%> (ΓΈ)
tests/unit/command/test_experiments.py 99.60% <100.00%> (+0.01%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

β˜” View full report at Codecov.
πŸ“’ Do you have feedback about the report comment? Let us know in this issue.

@pmrowla pmrowla merged commit 78fb527 into treeverse:main Mar 2, 2023
@pmrowla pmrowla deleted the exp-clean branch March 2, 2023 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A: experiments Related to dvc exp performance improvement over resource / time consuming tasks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Running exp queue status is very slow

2 participants