Cache hashes for --generate-hashes

[justicz]

Improves the speed of pip-compile --generate-hashes by caching the hashes we already have.

If hashes for a particular dependency have already been generated, and are available in the output file, this PR skips pulling all of the packages again from PyPI, dramatically increasing the speed of pip-compile when the --generate-hashes flag is passed.

This has the added benefit of improving security; if pip-compile --generate-hashes is run regularly as part of a build pipeline, hashes will be guaranteed stay the same unless the version has been explicitly updated.

Helps resolve #521

Maintainer Edit - Changelog:
Improved the speed of pip-compile --generate-hashes by caching the hashes from an existing output file.

[vphilippon]

Tested: Works and improved speed greatly with --generate-hashes.
LGTM 👍

[vphilippon]

Thanks for that first contribution @justicz, nice work!

[graingert]

does this include the new hash of the new wheel in the case that someone uploads a .tar.gz, then later a wheel?

[justicz]

This change doesn’t handle the case where new distributions are uploaded later for the same version. Is that going to be a problem? It’s definitely a good observation, but IMO that sounds like it will be a pretty uncommon issue.

[graingert]

I often hit this scenario. Because I regularly find packages without wheels and I go and poke the maintainer to go release them

[justicz]

OK. In that particular case, would it be acceptable for you to just delete the hashes in the generated output file for that package? Then they'll just be regenerated the usual, slow way. I would tend to think that this change provides the better UX 99% of the time.

[vphilippon]

A pip-compile with an existing requirements.txt would not have updated the versions (pins) unless required to, or explicitly told to do so with --upgrade or --upgrade-package. I think this PR actually aligns --generate-hashes with that: No changes unless required or explicitly told to do so.
Now we might want to add and option analog to --upgrade/--upgrade-package regarding hashes. Maybe something like --regenerate-hashes. Manual edits to the compiled requirements.txt can lead to a lot of trouble, and I usually advise against it (warranty void if touched 😛 ) .

(btw @graingert thanks for the observation, and for poking maintainers for wheels too 😄 )

[justicz]

A pip-compile with an existing requirements.txt would not have updated the versions (pins) unless required to, or explicitly told to do so with --upgrade or --upgrade-package.

Good to know! My mistake, I guess the security benefit I mentioned earlier is actually already the default behavior then?

Edit: Oh, never mind -- I missed that your comment was talking about versions and hash pins separately. In that case, my point about increasing security by not updating hashes still stands I think.

[vphilippon]

@justicz Your point on increased security sounds right to me, but don't quote me on this, I don't consider myself enough of a security guy to argue that it's better or worse in terms of security.
What I can say though is that I prefer this new behavior regarding how and when the requirements.txt and its content gets to be modified by a pip-compile.