Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installation freezes with Twisted[http2] #5059

Closed
elacuesta opened this issue Mar 21, 2021 · 5 comments · Fixed by #5066
Closed

Installation freezes with Twisted[http2] #5059

elacuesta opened this issue Mar 21, 2021 · 5 comments · Fixed by #5066
Labels
Milestone

Comments

@elacuesta
Copy link
Member

elacuesta commented Mar 21, 2021

I'm observing an considerable increase in the installation time after the addition of the Twisted[http2] dependency, most likely due to dependency backtracking in the new pip resolver. Actually not just an increase in time, I haven't been able to finish an installation after the addition. Also, my computer fan goes crazy, something it rarely does. I'm trying fresh Python3.8 virtualenvs each time.

I suspect this is not currently happening on master because we're not upgrading pip before running the tests, I opened #5058 to see what happens if we do.

Reasonable installation without Twisted[http2] (10.73s user 2.67s system 74% cpu 18.113 total)
$ python3.8 -m venv venv-scrapy-no-http2

$ source venv-scrapy-no-http2/bin/activate

$ pip install -U pip --no-cache-dir
Collecting pip
  Downloading pip-21.0.1-py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 5.4 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.1
    Uninstalling pip-20.2.1:
      Successfully uninstalled pip-20.2.1
Successfully installed pip-21.0.1

$ git log -1 --oneline
0e7ee125 (HEAD -> master, upstream/master) Merge pull request #5036 from dmiwell/urllength-dont-skip-silently

$ git diff setup.py
diff --git setup.py setup.py
index 767c6f6b..da2a8df9 100644
--- setup.py
+++ setup.py
@@ -19,7 +19,7 @@ def has_environment_marker_platform_impl_support():


 install_requires = [
-    'Twisted[http2]>=17.9.0',
+    'Twisted>=17.9.0',
     'cryptography>=2.0',
     'cssselect>=0.9.1',
     'itemloaders>=1.0.1',

$ time pip install --no-cache-dir -e .
Obtaining file:///Users/eus/zyte/scrapy
Collecting Twisted>=17.9.0
  Downloading Twisted-21.2.0-py3-none-any.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 4.9 MB/s
Collecting cryptography>=2.0
  Downloading cryptography-3.4.6-cp36-abi3-macosx_10_10_x86_64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 7.3 MB/s
Collecting cssselect>=0.9.1
  Downloading cssselect-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting itemloaders>=1.0.1
  Downloading itemloaders-1.0.4-py3-none-any.whl (11 kB)
Collecting parsel>=1.5.0
  Downloading parsel-1.6.0-py2.py3-none-any.whl (13 kB)
Collecting pyOpenSSL>=16.2.0
  Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 12.9 MB/s
Collecting queuelib>=1.4.2
  Downloading queuelib-1.5.0-py2.py3-none-any.whl (13 kB)
Collecting service_identity>=16.0.0
  Downloading service_identity-18.1.0-py2.py3-none-any.whl (11 kB)
Collecting w3lib>=1.17.0
  Downloading w3lib-1.22.0-py2.py3-none-any.whl (20 kB)
Collecting zope.interface>=4.1.3
  Downloading zope.interface-5.2.0-cp38-cp38-macosx_10_9_x86_64.whl (194 kB)
     |████████████████████████████████| 194 kB 10.8 MB/s
Collecting protego>=0.1.15
  Downloading Protego-0.1.16.tar.gz (3.2 MB)
     |████████████████████████████████| 3.2 MB 9.0 MB/s
Collecting itemadapter>=0.1.0
  Downloading itemadapter-0.2.0-py3-none-any.whl (9.3 kB)
Collecting h2>=3.2.0
  Downloading h2-4.0.0-py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 11.0 MB/s
Collecting lxml>=3.5.0
  Downloading lxml-4.6.2-cp38-cp38-macosx_10_9_x86_64.whl (4.6 MB)
     |████████████████████████████████| 4.6 MB 25.2 MB/s
Collecting PyDispatcher>=2.0.5
  Downloading PyDispatcher-2.0.5.zip (47 kB)
     |████████████████████████████████| 47 kB 8.7 MB/s
Collecting cffi>=1.12
  Downloading cffi-1.14.5-cp38-cp38-macosx_10_9_x86_64.whl (176 kB)
     |████████████████████████████████| 176 kB 8.8 MB/s
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 9.1 MB/s
Collecting hyperframe<7,>=6.0
  Downloading hyperframe-6.0.0-py3-none-any.whl (11 kB)
Collecting hpack<5,>=4.0
  Downloading hpack-4.0.0-py3-none-any.whl (32 kB)
Collecting jmespath>=0.9.5
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting six>=1.6.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting attrs>=16.0.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 22.8 MB/s
Collecting pyasn1-modules
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 8.0 MB/s
Collecting pyasn1
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 11.9 MB/s
Collecting hyperlink>=17.1.1
  Downloading hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
     |████████████████████████████████| 74 kB 9.7 MB/s
Collecting constantly>=15.1
  Downloading constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting Automat>=0.8.0
  Downloading Automat-20.2.0-py2.py3-none-any.whl (31 kB)
Collecting incremental>=16.10.1
  Downloading incremental-21.3.0-py2.py3-none-any.whl (15 kB)
Collecting idna>=2.5
  Downloading idna-3.1-py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 15.4 MB/s
Requirement already satisfied: setuptools in ./venv-scrapy-no-http2/lib/python3.8/site-packages (from zope.interface>=4.1.3->Scrapy==2.4.1) (49.2.1)
Using legacy 'setup.py install' for protego, since package 'wheel' is not installed.
Using legacy 'setup.py install' for PyDispatcher, since package 'wheel' is not installed.
Installing collected packages: six, pycparser, w3lib, pyasn1, lxml, idna, cssselect, cffi, attrs, zope.interface, pyasn1-modules, parsel, jmespath, itemadapter, incremental, hyperlink, hyperframe, hpack, cryptography, constantly, Automat, Twisted, service-identity, queuelib, pyOpenSSL, PyDispatcher, protego, itemloaders, h2, Scrapy
    Running setup.py install for PyDispatcher ... done
    Running setup.py install for protego ... done
  Running setup.py develop for Scrapy
Successfully installed Automat-20.2.0 PyDispatcher-2.0.5 Scrapy Twisted-21.2.0 attrs-20.3.0 cffi-1.14.5 constantly-15.1.0 cryptography-3.4.6 cssselect-1.1.0 h2-4.0.0 hpack-4.0.0 hyperframe-6.0.0 hyperlink-21.0.0 idna-3.1 incremental-21.3.0 itemadapter-0.2.0 itemloaders-1.0.4 jmespath-0.10.0 lxml-4.6.2 parsel-1.6.0 protego-0.1.16 pyOpenSSL-20.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 queuelib-1.5.0 service-identity-18.1.0 six-1.15.0 w3lib-1.22.0 zope.interface-5.2.0
pip install --no-cache-dir -e .  10.73s user 2.67s system 74% cpu 18.113 total
Ridiculous installation with Twisted[http2] (959.70s user 11.06s system 97% cpu 16:31.00 total, after cancellation!)
$ python3.8 -m venv venv-scrapy-http2

$ source venv-scrapy-http2/bin/activate

$ pip install --no-cache-dir -U pip
Collecting pip
   Downloading pip-21.0.1-py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 4.7 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.1
     Uninstalling pip-20.2.1:
      Successfully uninstalled pip-20.2.1
Successfully installed pip-21.0.1

$ git log -1 --oneline
0e7ee125 (HEAD -> master, upstream/master) Merge pull request #5036 from dmiwell/urllength-dont-skip-silently

$ git diff setup.py

$ time pip install --no-cache-dir -e .
Obtaining file:///Users/eus/zyte/scrapy
Collecting Twisted[http2]>=17.9.0
  Downloading Twisted-21.2.0-py3-none-any.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 5.0 MB/s
Collecting cryptography>=2.0
  Downloading cryptography-3.4.6-cp36-abi3-macosx_10_10_x86_64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 9.5 MB/s
Collecting cssselect>=0.9.1
  Downloading cssselect-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting itemloaders>=1.0.1
  Downloading itemloaders-1.0.4-py3-none-any.whl (11 kB)
Collecting parsel>=1.5.0
  Downloading parsel-1.6.0-py2.py3-none-any.whl (13 kB)
Collecting pyOpenSSL>=16.2.0
  Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 10.2 MB/s
Collecting queuelib>=1.4.2
  Downloading queuelib-1.5.0-py2.py3-none-any.whl (13 kB)
Collecting service_identity>=16.0.0
  Downloading service_identity-18.1.0-py2.py3-none-any.whl (11 kB)
Collecting w3lib>=1.17.0
  Downloading w3lib-1.22.0-py2.py3-none-any.whl (20 kB)
Collecting zope.interface>=4.1.3
  Downloading zope.interface-5.2.0-cp38-cp38-macosx_10_9_x86_64.whl (194 kB)
     |████████████████████████████████| 194 kB 8.2 MB/s
Collecting protego>=0.1.15
  Downloading Protego-0.1.16.tar.gz (3.2 MB)
     |████████████████████████████████| 3.2 MB 8.4 MB/s
Collecting itemadapter>=0.1.0
  Downloading itemadapter-0.2.0-py3-none-any.whl (9.3 kB)
Collecting h2>=3.2.0
  Downloading h2-4.0.0-py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 12.4 MB/s
Collecting lxml>=3.5.0
  Downloading lxml-4.6.2-cp38-cp38-macosx_10_9_x86_64.whl (4.6 MB)
     |████████████████████████████████| 4.6 MB 9.1 MB/s
Collecting PyDispatcher>=2.0.5
  Downloading PyDispatcher-2.0.5.zip (47 kB)
     |████████████████████████████████| 47 kB 24.6 MB/s
Collecting cffi>=1.12
  Downloading cffi-1.14.5-cp38-cp38-macosx_10_9_x86_64.whl (176 kB)
     |████████████████████████████████| 176 kB 8.0 MB/s
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 9.2 MB/s
Collecting hyperframe<7,>=6.0
  Downloading hyperframe-6.0.0-py3-none-any.whl (11 kB)
Collecting hpack<5,>=4.0
  Downloading hpack-4.0.0-py3-none-any.whl (32 kB)
Collecting jmespath>=0.9.5
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting six>=1.6.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting attrs>=16.0.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 10.0 MB/s
Collecting pyasn1
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 10.2 MB/s
Collecting pyasn1-modules
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 8.1 MB/s
Collecting Automat>=0.8.0
  Downloading Automat-20.2.0-py2.py3-none-any.whl (31 kB)
Collecting hyperlink>=17.1.1
  Downloading hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
     |████████████████████████████████| 74 kB 10.2 MB/s
Collecting constantly>=15.1
  Downloading constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting incremental>=16.10.1
  Downloading incremental-21.3.0-py2.py3-none-any.whl (15 kB)
Collecting h2>=3.2.0
  Downloading h2-3.2.0-py2.py3-none-any.whl (65 kB)
     |████████████████████████████████| 65 kB 12.3 MB/s
Collecting priority<2.0,>=1.1.0
  Downloading priority-1.3.0-py2.py3-none-any.whl (11 kB)
INFO: pip is looking at multiple versions of constantly to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of automat to determine which version is compatible with other requirements. This could take a while.
Collecting Automat>=0.8.0
  Downloading Automat-0.8.0-py2.py3-none-any.whl (31 kB)
INFO: pip is looking at multiple versions of twisted to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of twisted[http2] to determine which version is compatible with other requirements. This could take a while.
Collecting Twisted[http2]>=17.9.0
  Downloading Twisted-20.3.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 9.4 MB/s
  Downloading Twisted-19.10.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 30.9 MB/s
  Downloading Twisted-19.7.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 10.8 MB/s
  Downloading Twisted-19.2.1.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 9.4 MB/s
  Downloading Twisted-19.2.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 9.5 MB/s
  Downloading Twisted-18.9.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 23.1 MB/s
  Downloading Twisted-18.7.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 27.5 MB/s
  Downloading Twisted-18.4.0.tar.bz2 (3.0 MB)
     |████████████████████████████████| 3.0 MB 23.1 MB/s
  Downloading Twisted-17.9.0.tar.bz2 (3.0 MB)
     |████████████████████████████████| 3.0 MB 22.0 MB/s
Collecting PyHamcrest!=1.10.0,>=1.9.0
  Downloading PyHamcrest-2.0.2-py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 9.8 MB/s
Collecting Automat>=0.3.0
  Downloading Automat-0.7.0-py2.py3-none-any.whl (37 kB)
  Downloading Automat-0.6.0-py2.py3-none-any.whl (35 kB)
  Downloading Automat-0.5.0-py2.py3-none-any.whl (33 kB)
  Downloading Automat-0.4.0-py2.py3-none-any.whl (22 kB)
  Downloading Automat-0.3.0-py2.py3-none-any.whl (12 kB)
INFO: pip is looking at multiple versions of constantly to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of automat to determine which version is compatible with other requirements. This could take a while.
Collecting characteristic
  Downloading characteristic-14.3.0-py2.py3-none-any.whl (15 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of twisted to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of twisted[http2] to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of six to determine which version is compatible with other requirements. This could take a while.
Collecting six>=1.6.0
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.13.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.12.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.11.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.10.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.9.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.8.0-py2.py3-none-any.whl (9.7 kB)
  Downloading six-1.7.3-py2.py3-none-any.whl (9.5 kB)
  Downloading six-1.7.2-py2.py3-none-any.whl (9.6 kB)
  Downloading six-1.7.1-py2.py3-none-any.whl (9.6 kB)
  Downloading six-1.7.0-py2.py3-none-any.whl (9.3 kB)
  Downloading six-1.6.1-py2.py3-none-any.whl (8.6 kB)
  Downloading six-1.6.0-py2.py3-none-any.whl (8.5 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of six to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
Collecting attrs>=16.0.0
  Downloading attrs-20.2.0-py2.py3-none-any.whl (48 kB)
     |████████████████████████████████| 48 kB 13.2 MB/s
  Downloading attrs-20.1.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 17.6 MB/s
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
  Downloading attrs-19.2.0-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 14.7 MB/s
  Downloading attrs-19.1.0-py2.py3-none-any.whl (35 kB)
  Downloading attrs-18.2.0-py2.py3-none-any.whl (34 kB)
  Downloading attrs-18.1.0-py2.py3-none-any.whl (28 kB)
  Downloading attrs-17.4.0-py2.py3-none-any.whl (31 kB)
  Downloading attrs-17.3.0-py2.py3-none-any.whl (29 kB)
  Downloading attrs-17.2.0-py2.py3-none-any.whl (24 kB)
  Downloading attrs-17.1.0-py2.py3-none-any.whl (26 kB)
  Downloading attrs-16.3.0-py2.py3-none-any.whl (21 kB)
  Downloading attrs-16.2.0-py2.py3-none-any.whl (19 kB)
  Downloading attrs-16.1.0-py2.py3-none-any.whl (19 kB)
  Downloading attrs-16.0.0-py2.py3-none-any.whl (15 kB)
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of service-identity to determine which version is compatible with other requirements. This could take a while.
Collecting service_identity>=16.0.0
  Downloading service_identity-17.0.0-py2.py3-none-any.whl (13 kB)
  Downloading service_identity-16.0.0-py2.py3-none-any.whl (11 kB)
INFO: pip is looking at multiple versions of queuelib to determine which version is compatible with other requirements. This could take a while.
Collecting queuelib>=1.4.2
  Downloading queuelib-1.4.2-py2.py3-none-any.whl (10 kB)
^CERROR: Operation cancelled by user
pip install --no-cache-dir -e .  959.70s user 11.06s system 97% cpu 16:31.00 total

Right now I don't remember why we decided to add the dependency to setup.py instead of making it optional, should we reconsider that decision? Or maybe pin a specific version?

@elacuesta
Copy link
Member Author

38 minutes is a new record. This time it reached a few more packages but I cancelled when my computer started to sound like an airplane about to take off.

Cancelled installation
$ time pip install --no-cache-dir -e . 
Obtaining file:///Users/eus/zyte/scrapy
Collecting Twisted[http2]>=17.9.0
  Downloading Twisted-21.2.0-py3-none-any.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 5.0 MB/s 
Collecting cryptography>=2.0
  Downloading cryptography-3.4.6-cp36-abi3-macosx_10_10_x86_64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 8.7 MB/s 
Collecting cssselect>=0.9.1
  Downloading cssselect-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting itemloaders>=1.0.1
  Downloading itemloaders-1.0.4-py3-none-any.whl (11 kB)
Collecting parsel>=1.5.0
  Downloading parsel-1.6.0-py2.py3-none-any.whl (13 kB)
Collecting pyOpenSSL>=16.2.0
  Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 13.0 MB/s 
Collecting queuelib>=1.4.2
  Downloading queuelib-1.5.0-py2.py3-none-any.whl (13 kB)
Collecting service_identity>=16.0.0
  Downloading service_identity-18.1.0-py2.py3-none-any.whl (11 kB)
Collecting w3lib>=1.17.0
  Downloading w3lib-1.22.0-py2.py3-none-any.whl (20 kB)
Collecting zope.interface>=4.1.3
  Downloading zope.interface-5.2.0-cp38-cp38-macosx_10_9_x86_64.whl (194 kB)
     |████████████████████████████████| 194 kB 8.6 MB/s 
Collecting protego>=0.1.15
  Downloading Protego-0.1.16.tar.gz (3.2 MB)
     |████████████████████████████████| 3.2 MB 10.4 MB/s 
Collecting itemadapter>=0.1.0
  Downloading itemadapter-0.2.0-py3-none-any.whl (9.3 kB)
Collecting h2>=3.2.0
  Downloading h2-4.0.0-py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 11.2 MB/s 
Collecting lxml>=3.5.0
  Downloading lxml-4.6.2-cp38-cp38-macosx_10_9_x86_64.whl (4.6 MB)
     |████████████████████████████████| 4.6 MB 8.3 MB/s 
Collecting PyDispatcher>=2.0.5
  Downloading PyDispatcher-2.0.5.zip (47 kB)
     |████████████████████████████████| 47 kB 21.4 MB/s 
Collecting cffi>=1.12
  Downloading cffi-1.14.5-cp38-cp38-macosx_10_9_x86_64.whl (176 kB)
     |████████████████████████████████| 176 kB 9.9 MB/s 
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 9.0 MB/s 
Collecting hyperframe<7,>=6.0
  Downloading hyperframe-6.0.0-py3-none-any.whl (11 kB)
Collecting hpack<5,>=4.0
  Downloading hpack-4.0.0-py3-none-any.whl (32 kB)
Collecting jmespath>=0.9.5
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting six>=1.6.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting pyasn1
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 9.6 MB/s 
Collecting attrs>=16.0.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 19.3 MB/s 
Collecting pyasn1-modules
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 8.6 MB/s 
Collecting constantly>=15.1
  Downloading constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting hyperlink>=17.1.1
  Downloading hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
     |████████████████████████████████| 74 kB 10.8 MB/s 
Collecting Automat>=0.8.0
  Downloading Automat-20.2.0-py2.py3-none-any.whl (31 kB)
Collecting incremental>=16.10.1
  Downloading incremental-21.3.0-py2.py3-none-any.whl (15 kB)
Collecting priority<2.0,>=1.1.0
  Downloading priority-1.3.0-py2.py3-none-any.whl (11 kB)
Collecting h2>=3.2.0
  Downloading h2-3.2.0-py2.py3-none-any.whl (65 kB)
     |████████████████████████████████| 65 kB 10.8 MB/s 
INFO: pip is looking at multiple versions of constantly to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of automat to determine which version is compatible with other requirements. This could take a while.
Collecting Automat>=0.8.0
  Downloading Automat-0.8.0-py2.py3-none-any.whl (31 kB)
INFO: pip is looking at multiple versions of twisted to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of twisted[http2] to determine which version is compatible with other requirements. This could take a while.
Collecting Twisted[http2]>=17.9.0
  Downloading Twisted-20.3.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 7.8 MB/s 
  Downloading Twisted-19.10.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 9.3 MB/s 
  Downloading Twisted-19.7.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 9.1 MB/s 
  Downloading Twisted-19.2.1.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 11.1 MB/s 
  Downloading Twisted-19.2.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 11.1 MB/s 
  Downloading Twisted-18.9.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 8.9 MB/s 
  Downloading Twisted-18.7.0.tar.bz2 (3.1 MB)
     |████████████████████████████████| 3.1 MB 9.1 MB/s 
  Downloading Twisted-18.4.0.tar.bz2 (3.0 MB)
     |████████████████████████████████| 3.0 MB 7.0 MB/s 
  Downloading Twisted-17.9.0.tar.bz2 (3.0 MB)
     |████████████████████████████████| 3.0 MB 12.5 MB/s 
Collecting PyHamcrest!=1.10.0,>=1.9.0
  Downloading PyHamcrest-2.0.2-py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 7.6 MB/s 
Collecting Automat>=0.3.0
  Downloading Automat-0.7.0-py2.py3-none-any.whl (37 kB)
  Downloading Automat-0.6.0-py2.py3-none-any.whl (35 kB)
  Downloading Automat-0.5.0-py2.py3-none-any.whl (33 kB)
  Downloading Automat-0.4.0-py2.py3-none-any.whl (22 kB)
  Downloading Automat-0.3.0-py2.py3-none-any.whl (12 kB)
INFO: pip is looking at multiple versions of constantly to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of automat to determine which version is compatible with other requirements. This could take a while.
Collecting characteristic
  Downloading characteristic-14.3.0-py2.py3-none-any.whl (15 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of twisted to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of twisted[http2] to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of six to determine which version is compatible with other requirements. This could take a while.
Collecting six>=1.6.0
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.13.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.12.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.11.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.10.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.9.0-py2.py3-none-any.whl (10 kB)
  Downloading six-1.8.0-py2.py3-none-any.whl (9.7 kB)
  Downloading six-1.7.3-py2.py3-none-any.whl (9.5 kB)
  Downloading six-1.7.2-py2.py3-none-any.whl (9.6 kB)
  Downloading six-1.7.1-py2.py3-none-any.whl (9.6 kB)
  Downloading six-1.7.0-py2.py3-none-any.whl (9.3 kB)
  Downloading six-1.6.1-py2.py3-none-any.whl (8.6 kB)
  Downloading six-1.6.0-py2.py3-none-any.whl (8.5 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of six to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
Collecting attrs>=16.0.0
  Downloading attrs-20.2.0-py2.py3-none-any.whl (48 kB)
     |████████████████████████████████| 48 kB 13.3 MB/s 
  Downloading attrs-20.1.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 12.6 MB/s 
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
  Downloading attrs-19.2.0-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 13.2 MB/s 
  Downloading attrs-19.1.0-py2.py3-none-any.whl (35 kB)
  Downloading attrs-18.2.0-py2.py3-none-any.whl (34 kB)
  Downloading attrs-18.1.0-py2.py3-none-any.whl (28 kB)
  Downloading attrs-17.4.0-py2.py3-none-any.whl (31 kB)
  Downloading attrs-17.3.0-py2.py3-none-any.whl (29 kB)
  Downloading attrs-17.2.0-py2.py3-none-any.whl (24 kB)
  Downloading attrs-17.1.0-py2.py3-none-any.whl (26 kB)
  Downloading attrs-16.3.0-py2.py3-none-any.whl (21 kB)
  Downloading attrs-16.2.0-py2.py3-none-any.whl (19 kB)
  Downloading attrs-16.1.0-py2.py3-none-any.whl (19 kB)
  Downloading attrs-16.0.0-py2.py3-none-any.whl (15 kB)
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of service-identity to determine which version is compatible with other requirements. This could take a while.
Collecting service_identity>=16.0.0
  Downloading service_identity-17.0.0-py2.py3-none-any.whl (13 kB)
  Downloading service_identity-16.0.0-py2.py3-none-any.whl (11 kB)
INFO: pip is looking at multiple versions of queuelib to determine which version is compatible with other requirements. This could take a while.
Collecting queuelib>=1.4.2
  Downloading queuelib-1.4.2-py2.py3-none-any.whl (10 kB)
INFO: pip is looking at multiple versions of pyopenssl to determine which version is compatible with other requirements. This could take a while.
Collecting pyOpenSSL>=16.2.0
  Downloading pyOpenSSL-20.0.0-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 12.5 MB/s 
  Downloading pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 11.8 MB/s 
  Downloading pyOpenSSL-19.0.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 15.1 MB/s 
  Downloading pyOpenSSL-18.0.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 10.8 MB/s 
  Downloading pyOpenSSL-17.5.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 10.9 MB/s 
  Downloading pyOpenSSL-17.4.0-py2.py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 11.4 MB/s 
  Downloading pyOpenSSL-17.3.0-py2.py3-none-any.whl (51 kB)
     |████████████████████████████████| 51 kB 8.6 MB/s 
  Downloading pyOpenSSL-17.2.0-py2.py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 7.7 MB/s 
  Downloading pyOpenSSL-17.1.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 13.5 MB/s 
  Downloading pyOpenSSL-17.0.0-py2.py3-none-any.whl (51 kB)
     |████████████████████████████████| 51 kB 7.8 MB/s 
  Downloading pyOpenSSL-16.2.0-py2.py3-none-any.whl (43 kB)
     |████████████████████████████████| 43 kB 12.8 MB/s 
INFO: pip is looking at multiple versions of service-identity to determine which version is compatible with other requirements. This could take a while.
^CERROR: Operation cancelled by user
pip install --no-cache-dir -e .  2283.70s user 16.85s system 98% cpu 38:45.59 total

@Gallaecio
Copy link
Member

I’m not against implementing a workaround on our end, giving how this is likely to affect our users negatively. But I think this is actually an upstream issue that ultimately needs to be resolved on pip’s side.

@elacuesta
Copy link
Member Author

elacuesta commented Mar 22, 2021

I didn't know that was already reported upstream, thanks for the link.
However, I'm still having issues installing on a fresh virtualenv, even with the downgraded version ("pip<20.3") you suggested at #5052. It might be that I'm the only one having this problem, but I'm worried that if we release this as it is, it might be un-installable 😅

Could you confirm you're having no issues making a fresh installation?

Full installation + error log with pip<20.3
$ git log -1
commit ec5a7918ec2d8888ba356f9e3295dcd1ee935884 (HEAD -> master, upstream/master, origin/master, origin/HEAD)
Author: Adrián Chaves <adrian@chaves.io>
Date:   Mon Mar 22 11:25:40 2021 +0100

    Include Content-Length in HTTP/1.1 responses (#5057)

$ python -m venv venv-scrapy

$ source venv-scrapy/bin/activate

$ pip install --upgrade "pip<20.3"
Collecting pip<20.3
  Using cached https://files.pythonhosted.org/packages/cb/28/91f26bd088ce8e22169032100d4260614fc3da435025ff389ef1d396a433/pip-20.2.4-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.2.4

$ pip install -e .
Obtaining file:///home/eugenio/zyte/scrapy
Collecting Twisted[http2]>=17.9.0
  Using cached Twisted-21.2.0-py3-none-any.whl (3.1 MB)
Collecting cryptography>=2.0
  Using cached cryptography-3.4.6-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
Collecting cssselect>=0.9.1
  Using cached cssselect-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting itemloaders>=1.0.1
  Using cached itemloaders-1.0.4-py3-none-any.whl (11 kB)
Collecting parsel>=1.5.0
  Using cached parsel-1.6.0-py2.py3-none-any.whl (13 kB)
Collecting pyOpenSSL>=16.2.0
  Using cached pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
Collecting queuelib>=1.4.2
  Using cached queuelib-1.5.0-py2.py3-none-any.whl (13 kB)
Collecting service_identity>=16.0.0
  Using cached service_identity-18.1.0-py2.py3-none-any.whl (11 kB)
Collecting w3lib>=1.17.0
  Using cached w3lib-1.22.0-py2.py3-none-any.whl (20 kB)
Collecting zope.interface>=4.1.3
  Using cached zope.interface-5.3.0-cp38-cp38-manylinux2010_x86_64.whl (255 kB)
Processing /home/eugenio/.cache/pip/wheels/91/64/36/bd0d11306cb22a78c7f53d603c7eb74ebb6c211703bc40b686/Protego-0.1.16-py3-none-any.whl
Collecting itemadapter>=0.1.0
  Using cached itemadapter-0.2.0-py3-none-any.whl (9.3 kB)
Collecting h2>=3.2.0
  Using cached h2-4.0.0-py3-none-any.whl (57 kB)
Collecting lxml>=3.5.0
  Using cached lxml-4.6.3-cp38-cp38-manylinux1_x86_64.whl (5.4 MB)
Processing /home/eugenio/.cache/pip/wheels/d1/d7/61/11b5b370ee487d38b5408ecb7e0257db9107fa622412cbe2ff/PyDispatcher-2.0.5-py3-none-any.whl
Collecting attrs>=19.2.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting Automat>=0.8.0
  Using cached Automat-20.2.0-py2.py3-none-any.whl (31 kB)
Collecting constantly>=15.1
  Using cached constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting hyperlink>=17.1.1
  Using cached hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
Collecting incremental>=16.10.1
  Using cached incremental-21.3.0-py2.py3-none-any.whl (15 kB)
Collecting priority<2.0,>=1.1.0; extra == "http2"
  Using cached priority-1.3.0-py2.py3-none-any.whl (11 kB)
Collecting cffi>=1.12
  Using cached cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl (411 kB)
Collecting jmespath>=0.9.5
  Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting six>=1.6.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting pyasn1
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools in ./venv-scrapy/lib/python3.8/site-packages (from zope.interface>=4.1.3->Scrapy==2.4.1) (41.2.0)
Collecting hyperframe<7,>=6.0
  Using cached hyperframe-6.0.0-py3-none-any.whl (11 kB)
Collecting hpack<5,>=4.0
  Using cached hpack-4.0.0-py3-none-any.whl (32 kB)
Collecting idna>=2.5
  Using cached idna-3.1-py3-none-any.whl (58 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Installing collected packages: attrs, zope.interface, six, Automat, constantly, idna, hyperlink, incremental, hyperframe, hpack, h2, priority, Twisted, pycparser, cffi, cryptography, cssselect, lxml, w3lib, parsel, itemadapter, jmespath, itemloaders, pyOpenSSL, queuelib, pyasn1, pyasn1-modules, service-identity, protego, PyDispatcher, Scrapy
  Running setup.py develop for Scrapy
Successfully installed Automat-20.2.0 PyDispatcher-2.0.5 Scrapy Twisted-21.2.0 attrs-20.3.0 cffi-1.14.5 constantly-15.1.0 cryptography-3.4.6 cssselect-1.1.0 h2-4.0.0 hpack-4.0.0 hyperframe-6.0.0 hyperlink-21.0.0 idna-3.1 incremental-21.3.0 itemadapter-0.2.0 itemloaders-1.0.4 jmespath-0.10.0 lxml-4.6.3 parsel-1.6.0 priority-1.3.0 protego-0.1.16 pyOpenSSL-20.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 queuelib-1.5.0 service-identity-18.1.0 six-1.15.0 w3lib-1.22.0 zope.interface-5.3.0
WARNING: You are using pip version 20.2.4; however, version 21.0.1 is available.
You should consider upgrading via the '/home/eugenio/zyte/scrapy/venv-scrapy/bin/python -m pip install --upgrade pip' command.

$ scrapy shell
Traceback (most recent call last):
  File "/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (h2 4.0.0 (/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages), Requirement.parse('h2<4.0,>=3.0; extra == "http2"'), {'Twisted'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/eugenio/zyte/scrapy/venv-scrapy/bin/scrapy", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3251, in <module>
    def _initialize_master_working_set():
  File "/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3234, in _call_aside
    f(*args, **kwargs)
  File "/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3263, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (h2 4.0.0 (/home/eugenio/zyte/scrapy/venv-scrapy/lib/python3.8/site-packages), Requirement.parse('h2<4.0,>=3.0; extra == "http2"'), {'Twisted'})

@Gallaecio
Copy link
Member

It looks like the problem with the old pip dependency resolver is that it fails to do the job, and installs the latest h2 even though Twisted[http2] requires h2<4. 1 less workaround available, and 1 more issue we might want to resolve, if we want to allow people to install with an old pip.

@Gallaecio
Copy link
Member

So, do we make it optional, ask uses to install Twisted[http2] on their own for HTTP/2 support in Scrapy?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants