Perf improvement for file copies #3135
Conversation
dvc/system.py
Outdated
| import speedcopy | ||
|
|
||
| return speedcopy.copyfile(src, dest) | ||
| else: |
There was a problem hiding this comment.
This else is not needed since there is a return above.
| "speedcopy>=2.0.1", | ||
| "pyfastcopy>=1.0.3", |
There was a problem hiding this comment.
Both packages are not available in conda, so this PR will break our conda package 🙁Would you be willing to submit and maintain those packages in conda? Another simpler option is to make these packages optional, so that our System.copy can work with or without those packages. So pypi would be faster than conda package.
There was a problem hiding this comment.
It is not a dealbreaker, as we could help out with those packages too, we have quite some experience with conda-forge already. Just asking. 🙂
There was a problem hiding this comment.
Also, these deps should be conditional, i.e. none are needed in Python 3.8/Linux
There was a problem hiding this comment.
Hi @efiop, I won't have enough time to maintain conda packages as well, I will let you guys make the call as to whether that is worthwhile - not forgetting that this code will potentially be deleted under Python 3.8.
I have updated the patch to fall back to standard code if packages are missing.
There was a problem hiding this comment.
@rxxg No worries. Thanks for the PR! We'll try to create and maintain those conda packages ourselves.
3 participants
❗ Have you followed the guidelines in the Contributing to DVC list?
📖 Check this box if this PR does not require documentation updates, or if it does and you have created a separate PR in dvc.org with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the dvc.org repo.
❌ Have you checked DeepSource, CodeClimate, and other sanity checks below? We consider their findings recommendatory and don't expect everything to be addressed. Please review them carefully and fix those that actually improve code or fix bugs.
Thank you for the contribution - we'll try to review it as soon as possible. 🙏