Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pipenv lock hangs. It really does. #3827

Open
pilkibun opened this issue Jul 7, 2019 · 10 comments

Comments

@pilkibun
Copy link

commented Jul 7, 2019

This is a followup to #2681, which was closed without addressing the issue for some users.
Also #3812, #3829, SO and someone's blog rant.

Summary
pipenv lock downloads every available artifact of installed packages and their dependencies. It does this to calculate their hashes, even when the artifact url includes the hash in a fragment. For some large packages, such as scipy, which have large dependencies and many artifacts per version, this behavior can result in unreasonably long delays for some users (893MB vs. 50MB download). It's also bad netiquette towards pypi.

Symptoms

  • @connormclaud noted that lock seems peculiarly sensitive to network conditions before it disappeared by itself.
  • @Pithikos complained of pipenv lock hanging when installing scipy
  • @jackiekazil was surprised that the problem later seemed to disappear of its own.

#2681 was closed by @techalchemy with a comment suggesting the delay is due to lengthy build times (which don't affect pipenv lock), but asked users to provide steps to reproduce.

note:
All the packages fetched have wheels.

Details

pipenv.lock calls Resolver.resolve() which enables all artifacts

"""
Finds acceptable hashes for all of the given InstallRequirements.
"""
with self.repository.allow_all_wheels():
return {ireq: self.repository.get_hashes(ireq) for ireq in ireqs}

For a common setup consisting of scipy, pandas and numpy, here's the list of artifacts:

Artifacts Queued for Hash Retrieval
https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl#sha256=3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c
https://files.pythonhosted.org/packages/dd/bf/4138e7bfb757de47d1f4b6994648ec67a51efe58fa907c1e11e350cddfca/six-1.12.0.tar.gz#sha256=d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73
https://files.pythonhosted.org/packages/ad/99/5b2e99737edeb28c71bcbec5b5dda19d0d9ef3ca3e92e3e925e7c0bb364c/python-dateutil-2.8.0.tar.gz#sha256=c89805f6f4d64db21ed966fda138f8a5ed7a4fdbc1a8ee329ce1b74e3c74da9e
https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl#sha256=7e6584c74aeed623791615e26efd690f29817a27c73085b78e4bad02493df2fb
https://files.pythonhosted.org/packages/28/c1/901d7b80bf7793352fd0b2e6a8688d44bf2cba8d8772b8ca25f71f129f77/scipy-1.3.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=4907040f62b91c2e170359c3d36c000af783f0fa1516a83d6c1517cde0af5340
https://files.pythonhosted.org/packages/94/b2/03c3f3a2e5d07014103370d49415eccaaae2c07c2b25164be6f8a76913c6/scipy-1.3.0-cp35-cp35m-manylinux1_i686.whl#sha256=1db9f964ed9c52dc5bd6127f0dd90ac89791daa690a5665cc01eae185912e1ba
https://files.pythonhosted.org/packages/14/49/8f13fa215e10a7ab0731cc95b0e9bb66cf83c6a98260b154cfbd0b55fb19/scipy-1.3.0-cp35-cp35m-manylinux1_x86_64.whl#sha256=adadeeae5500de0da2b9e8dd478520d0a9945b577b2198f2462555e68f58e7ef
https://files.pythonhosted.org/packages/7a/ed/877f676c9e736b62385d1220602ceee3ca5c574bbb95c582a6b74bf047e2/scipy-1.3.0-cp35-cp35m-win32.whl#sha256=03b1e0775edbe6a4c64effb05fff2ce1429b76d29d754aa5ee2d848b60033351
https://files.pythonhosted.org/packages/ff/c5/64e6312d301c77f2a7acb5e755238bb8ba57e93feaade41ed73334ae2768/scipy-1.3.0-cp35-cp35m-win_amd64.whl#sha256=a7695a378c2ce402405ea37b12c7a338a8755e081869bd6b95858893ceb617ae
https://files.pythonhosted.org/packages/81/ae/125c21f09b202c3009ad7d9fb0263fb7d6053813d4b67ccbbe4d65f7f53a/scipy-1.3.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=826b9f5fbb7f908a13aa1efd4b7321e36992f5868d5d8311c7b40cf9b11ca0e7
https://files.pythonhosted.org/packages/42/df/cf4801dc7205d5535ddbdc426daa64a6e2120fb312dbb01d27814c9400a9/scipy-1.3.0-cp36-cp36m-manylinux1_i686.whl#sha256=b283a76a83fe463c9587a2c88003f800e08c3929dfbeba833b78260f9c209785
https://files.pythonhosted.org/packages/72/4c/5f81e7264b0a7a8bd570810f48cd346ba36faedbd2ba255c873ad556de76/scipy-1.3.0-cp36-cp36m-manylinux1_x86_64.whl#sha256=db61a640ca20f237317d27bc658c1fc54c7581ff7f6502d112922dc285bdabee
https://files.pythonhosted.org/packages/0b/27/fda16e2b06d6b2202f3c83efc6dea67fbf96aa158d00cc29a8aa6d645e88/scipy-1.3.0-cp36-cp36m-win32.whl#sha256=409846be9d6bdcbd78b9e5afe2f64b2da5a923dd7c1cd0615ce589489533fdbb
https://files.pythonhosted.org/packages/9e/fd/9a995b7fc18c6c17ce570b3cfdabffbd2718e4f1830e94777c4fd66e1179/scipy-1.3.0-cp36-cp36m-win_amd64.whl#sha256=c19a7389ab3cd712058a8c3c9ffd8d27a57f3d84b9c91a931f542682bb3d269d
https://files.pythonhosted.org/packages/04/66/ec5f1283d6a290a9153881a896837487338c44639c1305cc59e1c7b69cc9/scipy-1.3.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=09d008237baabf52a5d4f5a6fcf9b3c03408f3f61a69c404472a16861a73917e
https://files.pythonhosted.org/packages/9a/38/9f5aa6814c7062318ba147cbfcddacea71ac15ddf69dced0765fe8fea39d/scipy-1.3.0-cp37-cp37m-manylinux1_i686.whl#sha256=a84c31e8409b420c3ca57fd30c7589378d6fdc8d155d866a7f8e6e80dec6fd06
https://files.pythonhosted.org/packages/5d/bd/c0feba81fb60e231cf40fc8a322ed5873c90ef7711795508692b1481a4ae/scipy-1.3.0-cp37-cp37m-manylinux1_x86_64.whl#sha256=c5ea60ece0c0c1c849025bfc541b60a6751b491b6f11dd9ef37ab5b8c9041921
https://files.pythonhosted.org/packages/be/cc/6f7842a4d9aa7f51155f849185631e1201df255742de84d724ac33bff723/scipy-1.3.0-cp37-cp37m-win32.whl#sha256=6c0543f2fdd38dee631fb023c0f31c284a532d205590b393d72009c14847f5b1
https://files.pythonhosted.org/packages/53/17/9dfd64540b6645fae581462ed2d1c8e680b7e946ca2789c5149693660392/scipy-1.3.0-cp37-cp37m-win_amd64.whl#sha256=10325f0ffac2400b1ec09537b7e403419dcd25d9fee602a44e8a32119af9079e
https://files.pythonhosted.org/packages/cb/97/361c8c6ceb3eb765371a702ea873ff2fe112fa40073e7d2b8199db8eb56e/scipy-1.3.0.tar.gz#sha256=c3bb4bd2aca82fb498247deeac12265921fe231502a6bc6edea3ee7fe6c40a7a
https://files.pythonhosted.org/packages/8f/0b/1a2c21bb69138337dc079841aa4a45e5b2fc7a4260c0907f5254fb08f02e/numpy-1.16.4-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=b5554368e4ede1856121b0dfa35ce71768102e4aa55e526cb8de7f374ff78722
https://files.pythonhosted.org/packages/f2/34/8de93582f74bf3b9a277054b436b9cf53128d7b84820bc6eb859d0afac74/numpy-1.16.4-cp27-cp27m-manylinux1_i686.whl#sha256=e8baab1bc7c9152715844f1faca6744f2416929de10d7639ed49555a85549f52
https://files.pythonhosted.org/packages/20/2c/4d64f1cd4d2170b91d24ae45725de837bd40c34c9c04c94255c0f51c513d/numpy-1.16.4-cp27-cp27m-manylinux1_x86_64.whl#sha256=2a04dda79606f3d2f760384c38ccd3d5b9bb79d4c8126b67aff5eb09a253763e
https://files.pythonhosted.org/packages/13/2d/0fa2e8de7022a4a39497f4a9e384b8b129dbcf5d1b059f1043e21f6f0a48/numpy-1.16.4-cp27-cp27m-win32.whl#sha256=94f5bd885f67bbb25c82d80184abbf7ce4f6c3c3a41fbaa4182f034bba803e69
https://files.pythonhosted.org/packages/a6/db/18770d6b8419188d56b8ddd9794cb34c2d9f1d272ed8b40fa1ee38a3ca06/numpy-1.16.4-cp27-cp27m-win_amd64.whl#sha256=7dc253b542bfd4b4eb88d9dbae4ca079e7bf2e2afd819ee18891a43db66c60c7
https://files.pythonhosted.org/packages/35/51/7eae9042f5904463cb27fea567afc15e90956bd4b7cba98ec1969e58f74a/numpy-1.16.4-cp27-cp27mu-manylinux1_i686.whl#sha256=0778076e764e146d3078b17c24c4d89e0ecd4ac5401beff8e1c87879043a0633
https://files.pythonhosted.org/packages/1f/c7/198496417c9c2f6226616cff7dedf2115a4f4d0276613bab842ec8ac1e23/numpy-1.16.4-cp27-cp27mu-manylinux1_x86_64.whl#sha256=b0348be89275fd1d4c44ffa39530c41a21062f52299b1e3ee7d1c61f060044b8
https://files.pythonhosted.org/packages/48/b4/266431019b3b2e0f343a4f98db31add8a5ce2d464e30cdd9deaca29a8751/numpy-1.16.4-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=52c40f1a4262c896420c6ea1c6fda62cf67070e3947e3307f5562bd783a90336
https://files.pythonhosted.org/packages/38/61/3704bcbedb6fbcef9b92fe66d08af2b4328d10f199251e9b7a6db71547dc/numpy-1.16.4-cp35-cp35m-manylinux1_i686.whl#sha256=141c7102f20abe6cf0d54c4ced8d565b86df4d3077ba2343b61a6db996cefec7
https://files.pythonhosted.org/packages/bb/ef/d5a21cbc094d3f4d5b5336494dbcc9550b70c766a8345513c7c24ed18418/numpy-1.16.4-cp35-cp35m-manylinux1_x86_64.whl#sha256=6e4f8d9e8aa79321657079b9ac03f3cf3fd067bf31c1cca4f56d49543f4356a5
https://files.pythonhosted.org/packages/58/13/f5e2b4057707b62457085d48f27cde6caa594bfa0254aceb29405fb8b5a4/numpy-1.16.4-cp35-cp35m-win32.whl#sha256=d79f18f41751725c56eceab2a886f021d70fd70a6188fd386e29a045945ffc10
https://files.pythonhosted.org/packages/b4/1b/36bd20a4a1f41729c406014974925598edaeca1ca2510a2843892329b2f1/numpy-1.16.4-cp35-cp35m-win_amd64.whl#sha256=14270a1ee8917d11e7753fb54fc7ffd1934f4d529235beec0b275e2ccf00333b
https://files.pythonhosted.org/packages/0f/c9/3526a357b6c35e5529158fbcfac1bb3adc8827e8809a6d254019d326d1cc/numpy-1.16.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=a89e188daa119ffa0d03ce5123dee3f8ffd5115c896c2a9d4f0dbb3d8b95bfa3
https://files.pythonhosted.org/packages/e4/ca/037f4d2b7788bd077af2bbe887f7225c74c5df8bab4824514d7decb8a904/numpy-1.16.4-cp36-cp36m-manylinux1_i686.whl#sha256=ec31fe12668af687b99acf1567399632a7c47b0e17cfb9ae47c098644ef36797
https://files.pythonhosted.org/packages/87/2d/e4656149cbadd3a8a0369fcd1a9c7d61cc7b87b3903b85389c70c989a696/numpy-1.16.4-cp36-cp36m-manylinux1_x86_64.whl#sha256=27e11c7a8ec9d5838bc59f809bfa86efc8a4fd02e58960fa9c49d998e14332d5
https://files.pythonhosted.org/packages/73/98/cecf557b7f3f1dfac93171392887e4f7a606d6867752311c56a30742d581/numpy-1.16.4-cp36-cp36m-win32.whl#sha256=dc2ca26a19ab32dc475dbad9dfe723d3a64c835f4c23f625c2b6566ca32b9f29
https://files.pythonhosted.org/packages/20/ed/e036d31a9b2c750f270cbb1cfc1c0f94ac78ae504eea7eec3267be4e294a/numpy-1.16.4-cp36-cp36m-win_amd64.whl#sha256=ad3399da9b0ca36e2f24de72f67ab2854a62e623274607e37e0ce5f5d5fa9166
https://files.pythonhosted.org/packages/6b/be/608b7f72b851472388eafc010a5d46dae5d41610d0ac5df4c98c2ed1b865/numpy-1.16.4-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=f58ac38d5ca045a377b3b377c84df8175ab992c970a53332fa8ac2373df44ff7
https://files.pythonhosted.org/packages/dd/40/27395e0ab15dbcc5015899f4cc4ecbb535864db17cfb3b9a5bae66a98ea7/numpy-1.16.4-cp37-cp37m-manylinux1_i686.whl#sha256=f12b4f7e2d8f9da3141564e6737d79016fe5336cc92de6814eba579744f65b0a
https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl#sha256=cbddc56b2502d3f87fda4f98d948eb5b11f36ff3902e17cb6cc44727f2200525
https://files.pythonhosted.org/packages/07/46/656c25b39fc152ea525eef14b641993624a6325a8ae815b200de57cff0bc/numpy-1.16.4-cp37-cp37m-win32.whl#sha256=3c26010c1b51e1224a3ca6b8df807de6e95128b0908c7e34f190e7775455b0ca
https://files.pythonhosted.org/packages/ce/61/be72eee50f042db3acf0b1fb86650ad36d6c0d9be9fc29f8505d3b9d6baa/numpy-1.16.4-cp37-cp37m-win_amd64.whl#sha256=dd9bcd4f294eb0633bb33d1a74febdd2b9018b8b8ed325f861fffcd2c7660bb8
https://files.pythonhosted.org/packages/d3/4b/f9f4b96c0b1ba43d28a5bdc4b64f0b9d3fbcf31313a51bc766942866a7c7/numpy-1.16.4.zip#sha256=7242be12a58fec245ee9734e625964b97cf7e3f2f7d016603f9e56660ce479c7
https://files.pythonhosted.org/packages/52/ff/912fe03a623a70bcf297d466013a0b4f4c68c3b60f86bf226682d061fc09/pandas-0.24.2-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=17916d818592c9ec891cbef2e90f98cc85e0f1e89ed0924c9b5220dc3209c846
https://files.pythonhosted.org/packages/34/cc/1911d56b9464de76f7ef34c6b1c66e82d3a394fd6d0925d5203e903d4eee/pandas-0.24.2-cp27-cp27m-win32.whl#sha256=42e5ad741a0d09232efbc7fc648226ed93306551772fc8aecc6dce9f0e676794
https://files.pythonhosted.org/packages/61/57/6c233cc63597c6aa6337e717bdeabf791e8b618e9c893922a223e4e41cf4/pandas-0.24.2-cp27-cp27m-win_amd64.whl#sha256=c9a4b7c55115eb278c19aa14b34fcf5920c8fe7797a09b7b053ddd6195ea89b3
https://files.pythonhosted.org/packages/95/dd/295a1fa20c0d6207ba8df5c53c9c2340ca370f6a230b173c89c479761526/pandas-0.24.2-cp27-cp27mu-manylinux1_i686.whl#sha256=5149a6db3e74f23dc3f5a216c2c9ae2e12920aa2d4a5b77e44e5b804a5f93248
https://files.pythonhosted.org/packages/db/83/7d4008ffc2988066ff37f6a0bb6d7b60822367dcb36ba5e39aa7801fda54/pandas-0.24.2-cp27-cp27mu-manylinux1_x86_64.whl#sha256=cc8fc0c7a8d5951dc738f1c1447f71c43734244453616f32b8aa0ef6013a5dfb
https://files.pythonhosted.org/packages/da/82/1bc41a30737b70863c9b50983ca0413aa47905d215892ee136de3217bf3b/pandas-0.24.2-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=17450e25ae69e2e6b303817bdf26b2cd57f69595d8550a77c308be0cd0fd58fa
https://files.pythonhosted.org/packages/b1/57/069982c126d22bbdb4de71912eec2c1d8d303149a0d7b17927797479a0ed/pandas-0.24.2-cp35-cp35m-manylinux1_i686.whl#sha256=366f30710172cb45a6b4f43b66c220653b1ea50303fbbd94e50571637ffb9167
https://files.pythonhosted.org/packages/74/24/0cdbf8907e1e3bc5a8da03345c23cbed7044330bb8f73bb12e711a640a00/pandas-0.24.2-cp35-cp35m-manylinux1_x86_64.whl#sha256=4e718e7f395ba5bfe8b6f6aaf2ff1c65a09bb77a36af6394621434e7cc813204
https://files.pythonhosted.org/packages/f2/7a/f2ed4fde495eb8d13dc595382c33f8aa2b58e0911ca4b12c1ca825872493/pandas-0.24.2-cp35-cp35m-win32.whl#sha256=8c872f7fdf3018b7891e1e3e86c55b190e6c5cee70cab771e8f246c855001296
https://files.pythonhosted.org/packages/b3/59/38c88e1b26779b287a82c3d7601ec42c15e4acef09196e870c4fe9b77bd4/pandas-0.24.2-cp35-cp35m-win_amd64.whl#sha256=a3352bacac12e1fc646213b998bce586f965c9d431773d9e91db27c7c48a1f7d
https://files.pythonhosted.org/packages/2a/67/0a59cb257c72bb837575ca0ddf5f0fe2a482e98209b7a1bed8cde68ddb46/pandas-0.24.2-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=d7b460bc316064540ce0c41c1438c416a40746fd8a4fb2999668bf18f3c4acf1
https://files.pythonhosted.org/packages/7f/99/9c508429078eb4e103e22b4a191d12f7a9ceccee8db7ff18266cbe84e6c9/pandas-0.24.2-cp36-cp36m-manylinux1_i686.whl#sha256=c1bd07ebc15285535f61ddd8c0c75d0d6293e80e1ee6d9a8d73f3f36954342d0
https://files.pythonhosted.org/packages/19/74/e50234bc82c553fecdbd566d8650801e3fe2d6d8c8d940638e3d8a7c5522/pandas-0.24.2-cp36-cp36m-manylinux1_x86_64.whl#sha256=071e42b89b57baa17031af8c6b6bbd2e9a5c68c595bc6bf9adabd7a9ed125d3b
https://files.pythonhosted.org/packages/65/3e/16260dcad8d28167f8622dd5e600700fa1665a9dc0b245bb6068a34f657c/pandas-0.24.2-cp36-cp36m-win32.whl#sha256=2538f099ab0e9f9c9d09bbcd94b47fd889bad06dc7ae96b1ed583f1dc1a7a822
https://files.pythonhosted.org/packages/d0/4e/9db3468e504ac9aeadb37eb32bcf0a74d063d24ad1471104bd8a7ba20c97/pandas-0.24.2-cp36-cp36m-win_amd64.whl#sha256=83c702615052f2a0a7fb1dd289726e29ec87a27272d775cb77affe749cca28f8
https://files.pythonhosted.org/packages/fc/43/fd867e3347559845c8f993059d410c50a1e18709f1c4d4b3b47323a06a37/pandas-0.24.2-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=627594338d6dd995cfc0bacd8e654cd9e1252d2a7c959449228df6740d737eb8
https://files.pythonhosted.org/packages/22/e6/2d47835f91eb010036be207581fa113fb4e3822ec1b4bafb0d3d105fede6/pandas-0.24.2-cp37-cp37m-manylinux1_x86_64.whl#sha256=4fe0d7e6438212e839fc5010c78b822664f1a824c0d263fd858f44131d9166e2
https://files.pythonhosted.org/packages/04/8b/a1d5f257bfee6aab440e763505c65368bbf713309c9dc70ceeb8d4677c2e/pandas-0.24.2-cp37-cp37m-win32.whl#sha256=bcdd06007cca02d51350f96debe51331dec429ac8f93930a43eb8fb5639e3eb5
https://files.pythonhosted.org/packages/61/c7/f943fceb712579bc538700e2c157dc4972e16abfe29bd4969149bad98c74/pandas-0.24.2-cp37-cp37m-win_amd64.whl#sha256=90f116086063934afd51e61a802a943826d2aac572b2f7d55caaac51c13db5b5
https://files.pythonhosted.org/packages/b2/4c/b6f966ac91c5670ba4ef0b0b5613b5379e3c7abdfad4e7b89a87d73bae13/pandas-0.24.2.tar.gz#sha256=4f919f409c433577a501e023943e582c57355d50a724c589e78bc1d551a535a2
https://files.pythonhosted.org/packages/3d/73/fe30c2daaaa0713420d0382b16fbb761409f532c56bdcc514bf7b6262bb6/pytz-2019.1-py2.py3-none-any.whl#sha256=303879e36b721603cc54604edcac9d20401bdbe31e1e4fdee5b9f98d5d31dfda
https://files.pythonhosted.org/packages/df/d5/3e3ff673e8f3096921b3f1b79ce04b832e0100b4741573154b72b756a681/pytz-2019.1.tar.gz#sha256=d747dd3d23d77ef44c6a3526e274af6efeb0a6f1afd5a69ba4d5be4098c8e141

That's a lot of sequential network round-trips to wait through, and most of these are for different platforms than the installation. But that's not all.

Each artifact is passed to HashCache.get_hash.

def get_hash(self, location):
# if there is no location hash (i.e., md5 / sha256 / etc) we on't want to store it
hash_value = None
vcs = VcsSupport()
orig_scheme = location.scheme
new_location = copy.deepcopy(location)
if orig_scheme in vcs.all_schemes:
new_location.url = new_location.url.split("+", 1)[-1]
can_hash = new_location.hash
if can_hash:
# hash url WITH fragment
hash_value = self.get(new_location.url)

The pypi artifact urls includes the sha256 hash, parsed into new_location.hash, but it isn't used. If the HashCache doesn't already hold the location key, the artifact is downloaded, hashed and then the hash is stored. This is the root cause of the delay, and in the above examples results in downloading 893MB of artifacts, compared to the 50MBs-worth which are installed.

The problem disappears if the user is patient enough to wait it out, or if the connection is fast. However, it seems most users are surprised by the delay (me included), and expect it to be more or less instantaneous once the packages are installed.

As a quick verification, I patched the get_hash method to use the hash fragment from the url if available, and the pipenv lock run time dropped to a few seconds.

Sidenote, --verbose does not log network requests, so this was masked from view even when trying to debug.

@pilkibun pilkibun changed the title pipenv lock downloads every possible artifact for every pyver/arch of all packages pipenv lock hangs. It really does. Jul 7, 2019

@frostming

This comment has been minimized.

Copy link
Collaborator

commented Jul 9, 2019

It seems a good solution, I'm +1 on this. Thanks for your efforts in it. Would you mind send a PR with the fix for this?

@jackiekazil

This comment has been minimized.

Copy link

commented Jul 9, 2019

I don't think I have ever felt so validated by a comment I left on github in hopes that one day in future someone would stumble upon it. (Turned out to be the near future.) <3 Thank you for this.

@pilkibun pilkibun referenced this issue Jul 9, 2019
2 of 2 tasks complete
@techalchemy

This comment has been minimized.

Copy link
Member

commented Jul 9, 2019

#2681 was closed by @techalchemy with a comment suggesting the delay is due to lengthy build times (which don't affect pipenv lock)

Build times absolutely affect pipenv lock, I am surprised if you investigated this you would conclude that they don't. pipenv lock resolves dependencies. In python, for anything that is not a wheel, that requires acquiring the artifact and either building it or parsing the AST of the setup.py file in question. There is simply no way around that. On slow internet connections that may not be the limiting factor, and with people building and releasing wheels more it is becoming less of an issue, but it is absolutely something that happens during locking and remains a significant contributor to the time it takes. At this point even pip builds artifacts in order to resolve dependencies simply for installation.

Dependency resolution is an NP hard problem, there is no hack or easy trick around this, and in python it is also a problem that sometimes requires building artifacts. If you have identified and can avoid extra downloads that is excellent, but I do want to be clear: building is often a part of locking.

@jackiekazil apologies if you felt invalidated by the previous responses, we are all definitely aware that locking is slow, and as I mentioned in the other threads on the topic I agree that there are likely multiple downloads occurring but am not precisely sure where and would need to see debugging info to make any progress, so if the accompanying PR here addresses that it is awesome

@tobiasfeil

This comment has been minimized.

Copy link

commented Jul 10, 2019

I'm going back to pip. This issue has been around for so long, and is the main blockage for new users. Makes the whole project seem a little bit unprofessional to me.

@iloveicedgreentea

This comment has been minimized.

Copy link

commented Jul 24, 2019

I have also gone back to pip. Every time an issue is opened, it is ignored. Pipenv, fundamentally, is unusable. Lock fails for me even with 10 relatively small packages. Its a shame because this was a great idea, only to be ruined by the locking mechanism.

@siben168

This comment has been minimized.

Copy link

commented Aug 6, 2019

my very first installation by pipenv got stuck by stopless "Locking", it just a small package and has already been install succeeded.

Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
[====] Locking...

1 hour later, it was still there... annoying.

@michio-nikaido

This comment has been minimized.

Copy link

commented Aug 14, 2019

(posting this in the hopes that some reports of hanging are due to the same issue I encountered)
I have been seeing issues with a variety of different pipenv commands, but most recently with 'pipenv lock' in a project that all my colleagues have had no issues with. This project had very few dependencies and I don't believe any of the 'large packages' (mentioned by others) were involved. On my laptop, I was able to consistently repro the 'hang' (actually it would timeout after 30 minutes) when running 'pipenv lock' while my colleagues reported success within a few seconds.

After multiple attempts to debug this, I kept seeing the same package causing problems (in my case, it was configparser, but I don't believe that's particularly relevant). I managed to replicate how pipenv runs resolver.py and recreated that environment in order to run the resolver.py under pdb. Under those conditions, I found that resolver.py would get stuck sitting in a sleep loop buried in a call stack that looked like it was trying to do some lockfile operations. When I looked at the directory it was trying to create in order to acquire the lock, it became clear what the problem was. The directory already existed and, in fact, had been on my filesystem for months (the length of time I have been having issues).

So in short, once I found:

~/Library/Caches/pipenv/http/f/5/9/3/d $ ll
total 8
17980163 drwx------  4 michio.nikaido  staff   136B May 28 11:59 ./
17980814 -rw-------  1 michio.nikaido  staff   3.6K May 28 11:59 f593d3690af2ee7367984e87eb3821fa2a514d37d808c806b4c7719a
17980812 drwxr-xr-x  2 michio.nikaido  staff    68B May 28 11:59 f593d3690af2ee7367984e87eb3821fa2a514d37d808c806b4c7719a.lock/
17980162 drwx------  3 michio.nikaido  staff   102B May 28 11:59 ../

I moved the that subtree to another location and then was able to successfully run pipenv lock again.

So for Mac OS, something like 'find ~/Library/Caches/pipenv/http | grep lock' might help highlight issues of this sort.

@Sraw

This comment has been minimized.

Copy link

commented Aug 21, 2019

Well, I really cannot understand this.
@allPipenvMembers Hey, everyone, this is a blocking issue but not a trivial issue. It DO DOES DID DONE IS a blocking issue. Which really prevents even fanatics from using pipenv.

Long-time ago, pypi doesn't contain metadata of the packages. There is no other choice but downloading all packages to calculate the hash. But it is 2019 now, pypi provides what you need.

In the past, I recommended this tool to others. Said "Oh, this is the next generation of pip.", "It is perfect, it not only manage virtual envs for you but also lock your dependencies.", "Just use it." ...
But now I really cannot open my mouth again.

@nourwolf

This comment has been minimized.

Copy link

commented Aug 22, 2019

Thanks @michio-nikaido
I confirm deleting the cache folders solved the issue for me.

@wolfgangpfnuer

This comment has been minimized.

Copy link

commented Sep 6, 2019

had the same problem. no error, verbose does nothing. really frustrating.
Fixed by putting the following into my make clean:
rm -rf ~/Library/Caches/pipenv/http/**/*.lock # fix weird pipenv bug
check for correct path for your OS by
python3 -c 'import appdirs; print(appdirs.user_cache_dir("pipenv"))'

the ** needs globstar
shopt -s globstar # allow **/*.py to work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.