Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable multithread for crate download/verify #7

Closed
wants to merge 1 commit into from

Conversation

hehaoqian
Copy link

This should significantly increase the download speed for crates.

Previously, It took days to download all crates
Now, it takes hours (tested with 64 threads)

@hehaoqian
Copy link
Author

@drmikehenry Modify this PR directly if you have found any problems. Thanks

This should significantly increase the download speed for crates
@drmikehenry
Copy link
Owner

Thanks for your pull request. I'd like to take a little time to mull it over and do some experimentation.

@drmikehenry
Copy link
Owner

Thanks for the suggestion to improve performance via parallel downloading. After some research, I have no confidence that the requests library is thread-safe (see, for example, the unclosed ticket psf/requests#2766 which merely asks for documentation to answer the question whether requests is thread-safe; this question has remained apparently unanswerable for nearly six years). Rather than venture down the multi-threading route, I've taken this as an opportunity to switch from requests to httpx and to implement asynchronous parallel downloading via the trio library. Feel free to open an issue if you discover any problems with this new implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants