Skip to content

Conversation

@karajan1001
Copy link
Contributor

@karajan1001 karajan1001 commented Jan 30, 2023

fixes: #8762
partly solves: #8477 and #8787
Improve the performance

Wait for treeverse/scmrepo#169

The performance improvement are shown in treeverse/scmrepo#169.

For the benchmark because the fetch operation only happens during temp or queue experiment running progress. We don't have this kind of benchmark and also hard to build one.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

@karajan1001 karajan1001 added performance improvement over resource / time consuming tasks A: experiments Related to dvc exp git Related to git and git backends labels Jan 30, 2023
@karajan1001 karajan1001 requested a review from pmrowla January 30, 2023 13:25
@karajan1001 karajan1001 self-assigned this Jan 30, 2023
@karajan1001 karajan1001 force-pushed the use_pygit2_for_fetch_refspec branch 2 times, most recently from 4068b7e to b220775 Compare February 2, 2023 08:25
@codecov
Copy link

codecov bot commented Feb 2, 2023

Codecov Report

Base: 93.12% // Head: 93.33% // Increases project coverage by +0.21% 🎉

Coverage data is based on head (49a27f4) compared to base (97589a1).
Patch has no changes to coverable lines.

❗ Current head 49a27f4 differs from pull request most recent head 4b9fa61. Consider uploading reports for the commit 4b9fa61 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8907      +/-   ##
==========================================
+ Coverage   93.12%   93.33%   +0.21%     
==========================================
  Files         453      453              
  Lines       36600    36495     -105     
  Branches     5287     5278       -9     
==========================================
- Hits        34083    34062      -21     
+ Misses       2001     1917      -84     
  Partials      516      516              
Impacted Files Coverage Δ
dvc/repo/update.py 48.64% <0.00%> (-2.87%) ⬇️
dvc/output.py 88.98% <0.00%> (ø)
dvc/config_schema.py 94.28% <0.00%> (ø)
tests/unit/remote/test_webdav.py 100.00% <0.00%> (ø)
dvc/repo/worktree.py 10.06% <0.00%> (+0.01%) ⬆️
dvc/testing/remote_tests.py 50.65% <0.00%> (+6.83%) ⬆️
dvc/testing/workspace_tests.py 73.85% <0.00%> (+7.18%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@karajan1001 karajan1001 force-pushed the use_pygit2_for_fetch_refspec branch 3 times, most recently from 4b9fa61 to 15c7956 Compare February 3, 2023 08:12
@efiop
Copy link
Contributor

efiop commented Feb 3, 2023

For the benchmark because the fetch operation only happens during temp or queue experiment running progress. We don't have this kind of benchmark and also hard to build one.

Are there any manual results?

@karajan1001
Copy link
Contributor Author

karajan1001 commented Feb 6, 2023

For the benchmark because the fetch operation only happens during temp or queue experiment running progress. We don't have this kind of benchmark and also hard to build one.

Are there any manual results?

The performance improvement are shown in treeverse/scmrepo#169.

This is from original.
image
This is after using pygit2 backend
image

@karajan1001 karajan1001 force-pushed the use_pygit2_for_fetch_refspec branch from 15c7956 to 5fd7ca4 Compare February 6, 2023 08:31
fix: treeverse#8762
1. partly solve treeverse#8787 and treeverse#8477
2. Improve the performance
3. Bump `scmrepo` into 0.1.9
@karajan1001 karajan1001 force-pushed the use_pygit2_for_fetch_refspec branch from 5fd7ca4 to 399176b Compare February 6, 2023 09:46
@efiop efiop merged commit e1acab5 into treeverse:main Feb 6, 2023
@karajan1001 karajan1001 deleted the use_pygit2_for_fetch_refspec branch February 6, 2023 14:07
@efiop
Copy link
Contributor

efiop commented Feb 6, 2023

Great work, @karajan1001 ! 🔥

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A: experiments Related to dvc exp git Related to git and git backends performance improvement over resource / time consuming tasks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

queue: checkpoint experiments hanging

3 participants