Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accelerate locking by fetching hash from url fragment #4500

Merged
merged 12 commits into from
Oct 28, 2020
Merged

Conversation

frostming
Copy link
Contributor

@frostming frostming commented Oct 27, 2020

Thank you for contributing to Pipenv!

The issue

Close #2681
Close #1914
Close #3827
Close #2873
Close #4430

The fix

Previously, package hash values are calculated based on the file content, so all matched wheels and sdists need to be actually downloaded. For packages like tensorflow and numpy, this is a great overhead to lock dependencies, whose average file size is ~200MB.

With this patch, Pipenv will retrieve hash value directly from the URL fragment when possible.

The checklist

  • Associated issue
  • A news fragment in the news/ directory to describe this fix with the extension .bugfix, .feature, .behavior, .doc. .vendor. or .trivial (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

@frostming frostming added Category: Dependency Resolution Issue relates to dependency resolution. Category: Performance Issue relates to performance labels Oct 27, 2020
@uranusjr
Copy link
Member

Do we want to submit this to upstream pip-tools as well?

@frostming
Copy link
Contributor Author

let me check, since the whole logic is added by pipenv

@frostming
Copy link
Contributor Author

@uranusjr Confirmed, the latest version of piptools is already reading hashes from PyPI JSON API. So I would like to keep this change in Pipenv only

@uranusjr
Copy link
Member

Cool. I’m curious why they chose to use the JSON API, but 🤷

@frostming frostming merged commit 912164d into master Oct 28, 2020
@frostming frostming deleted the url-hash branch October 28, 2020 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category: Dependency Resolution Issue relates to dependency resolution. Category: Performance Issue relates to performance
Projects
None yet
2 participants