pip resolver fails in a fresh env when installing jupyter with other stuff

[amitbeka]

Hi there,

(got here through the call-for-reports, so this is not a pressing issue)

I tried pip resolver and it failed with the following setup, in a fresh pyenv virtualenv:

> python --version
Python 3.8.2
> pip install -U pip wheel
> pip -V
pip 20.1.1 from /home/abeka/.pyenv/versions/3.8.2/envs/temp/lib/python3.8/site-packages/pip (python 3.8)

Then tried to install the next requirements file:

boto
boto3
click
concurrentloghandler
django>=1.11,<1.11.99
dulwich
future
fuzzywuzzy
genty
graphyte
gunicorn
inflect
jellyfish
numpy
pandas
pynlpl
python-levenshtein
pyyaml
regex
requests>=2.20
supervisor
tqdm
uwsgi
uwsgidecorators
awscli
black
boto3
cmd2
datadiff
genty
pandas
pip-tools
pyflakes
pylint-django
pylint<2.5
virtualenv
virtualenvwrapper
jupyter
pudb

The command used:

pip install --unstable-feature=resolver -r requirements.txt

And got this output with the pip._vendor.resolvelib.resolvers.ResolutionTooDeep exception:

Collecting https://files.pythonhosted.org/packages/c3/32/a0fa1475ce5eed11f8381359ca2b186578be7096e1b02628e717dd3231eb/uwsgidecorators-1.1.0.tar.gz#sha256=92693eeed4836fa3afde2954629bd9f846ea358b965fba6a00ce365cce68f976 (from -r requirements.txt (line 24))
  Using cached uwsgidecorators-1.1.0.tar.gz (3.2 kB)
Collecting jupyter==1.0.0
  Using cached jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB)
Collecting https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz#sha256=033a11de5e3d19ea25c9302d11224e1a1898fe5abd23c61c7c360c25195e3eb1 (from -r requirements.txt (line 17))
  Using cached python-Levenshtein-0.12.0.tar.gz (48 kB)
Requirement already satisfied: setuptools==41.2.0 in /home/abeka/.pyenv/versions/3.8.2/envs/temp/lib/python3.8/site-packages (from python-Levenshtein==0.12.0->-r requirements.txt (line 17)) (41.2.0)
Collecting https://files.pythonhosted.org/packages/29/83/6c63b792f822aa439165596d288373911a13e5c9fe6c79043a8727448144/graphyte-1.6.0.tar.gz#sha256=ab1fd5a0061cd8f412d379273e36dfb4b80906c4c804052cfe3aa1ed416dc072 (from -r requirements.txt (line 10))
  Using cached graphyte-1.6.0.tar.gz (7.3 kB)
Collecting https://files.pythonhosted.org/packages/87/72/74d5c3d9b4574d3df45f61894aa28344b22f4b059d192cc3753c112f62f0/datadiff-2.0.0.tar.gz#sha256=54049440a896de5247ee2e7b7aa5425f939ac832c37edb2101f8d3036e89caa8 (from -r requirements.txt (line 29))
  Using cached datadiff-2.0.0.tar.gz (5.5 kB)
Collecting requests==2.23.0
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting chardet==3.0.4
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna==2.9
  Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting genty==1.3.2
  Using cached genty-1.3.2-py2.py3-none-any.whl (19 kB)
Collecting https://files.pythonhosted.org/packages/fd/e5/0dc4f256bcc6484d454006b02f33263b20f762a433741b29d53875e0d763/ConcurrentLogHandler-0.9.1.tar.gz#sha256=8225a590fd4194c413fa26675bde5f6b80ad79e4182d5876ba3e264f77755918 (from -r requirements.txt (line 4))
  Using cached ConcurrentLogHandler-0.9.1.tar.gz (25 kB)
Collecting https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz#sha256=b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d (from -r requirements.txt (line 18))
  Using cached PyYAML-5.3.1.tar.gz (269 kB)
Collecting https://files.pythonhosted.org/packages/3f/80/bcacc7affb47be7279d7d35225e1a932416ed051b315a7f9df20acf04cbe/jellyfish-0.7.2.tar.gz#sha256=cb09c50d7e2bb7b926fc7654762bc81f9c629e0c92ae7137bf22b34f39515286 (from -r requirements.txt (line 13))
  Using cached jellyfish-0.7.2.tar.gz (133 kB)
Collecting jupyter-console==6.1.0
  Using cached jupyter_console-6.1.0-py2.py3-none-any.whl (21 kB)
Collecting prompt-toolkit==3.0.5
  Using cached prompt_toolkit-3.0.5-py3-none-any.whl (351 kB)
Collecting wcwidth==0.1.9
  Using cached wcwidth-0.1.9-py2.py3-none-any.whl (19 kB)
Collecting urllib3==1.25.9
  Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Collecting certifi==2020.4.5.1
  Using cached certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
Collecting supervisor==4.2.0
  Using cached supervisor-4.2.0-py2.py3-none-any.whl (738 kB)
Collecting nbconvert==5.6.1
  Using cached nbconvert-5.6.1-py2.py3-none-any.whl (455 kB)
Collecting https://files.pythonhosted.org/packages/4c/ea/236e2584af67bb6df960832731a6e5325fd4441de001767da328c33368ce/pandocfilters-1.4.2.tar.gz#sha256=b3dd70e169bb5449e6bc6ff96aea89c5eea8c5f6ab5e207fc2f521a2cf4a0da9 (from nbconvert==5.6.1->jupyter==1.0.0->-r requirements.txt (line 38))
  Using cached pandocfilters-1.4.2.tar.gz (14 kB)
Collecting entrypoints==0.3
  Using cached entrypoints-0.3-py2.py3-none-any.whl (11 kB)
Collecting mistune==0.8.4
  Using cached mistune-0.8.4-py2.py3-none-any.whl (16 kB)
Collecting defusedxml==0.6.0
  Using cached defusedxml-0.6.0-py2.py3-none-any.whl (23 kB)
Collecting nbformat==5.0.6
  Using cached nbformat-5.0.6-py3-none-any.whl (170 kB)
Collecting ipython-genutils==0.2.0
  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting testpath==0.4.4
  Using cached testpath-0.4.4-py2.py3-none-any.whl (163 kB)
Collecting traitlets==4.3.3
  Using cached traitlets-4.3.3-py2.py3-none-any.whl (75 kB)
Collecting jsonschema==3.2.0
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting six==1.14.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting attrs==19.3.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting https://files.pythonhosted.org/packages/9f/0d/cbca4d0bbc5671822a59f270e4ce3f2195f8a899c97d0d5abb81b191efb5/pyrsistent-0.16.0.tar.gz#sha256=28669905fe725965daa16184933676547c5bb40a5153055a8dee2a4bd7933ad3 (from jsonschema==3.2.0->nbformat==5.0.6->nbconvert==5.6.1->jupyter==1.0.0->-r requirements.txt (line 38))
  Using cached pyrsistent-0.16.0.tar.gz (108 kB)
Collecting fuzzywuzzy==0.18.0
  Using cached fuzzywuzzy-0.18.0-py2.py3-none-any.whl (18 kB)
Collecting inflect==4.1.0
  Using cached inflect-4.1.0-py3-none-any.whl (31 kB)
Collecting black==19.10b0
  Using cached black-19.10b0-py36-none-any.whl (97 kB)
Collecting typed-ast==1.4.1
  Using cached typed_ast-1.4.1-cp38-cp38-manylinux1_x86_64.whl (768 kB)
Collecting pathspec==0.8.0
  Using cached pathspec-0.8.0-py2.py3-none-any.whl (28 kB)
Collecting toml==0.10.1
  Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting click==7.1.2
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting appdirs==1.4.4
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting jupyter-core==4.6.3
  Using cached jupyter_core-4.6.3-py2.py3-none-any.whl (83 kB)
Collecting Jinja2==2.11.2
  Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting MarkupSafe==1.1.1
  Using cached MarkupSafe-1.1.1-cp38-cp38-manylinux1_x86_64.whl (32 kB)
Collecting decorator==4.4.2
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting pyflakes==2.2.0
  Using cached pyflakes-2.2.0-py2.py3-none-any.whl (66 kB)
Collecting https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz#sha256=b1bead90b70cf6ec3f0710ae53a525360fa360d306a86583adc6bf83a4db537d (from -r requirements.txt (line 7))
  Using cached future-0.18.2.tar.gz (829 kB)
Collecting https://files.pythonhosted.org/packages/e7/1e/3dcca007f974fe4eb369bf1b8629d5e342bb3055e2001b2e5340aaefae7a/uwsgi-2.0.18.tar.gz#sha256=4972ac538800fb2d421027f49b4a1869b66048839507ccf0aa2fda792d99f583 (from -r requirements.txt (line 23))
  Using cached uwsgi-2.0.18.tar.gz (801 kB)
Collecting dulwich==0.19.16
  Downloading dulwich-0.19.16-py3-none-any.whl (428 kB)
     |████████████████████████████████| 428 kB 105 kB/s 
Collecting https://files.pythonhosted.org/packages/c1/6b/2f05d73b2d2f2410b48b90d3783a0034c26afa534a4a95ad5f1178d61191/virtualenvwrapper-4.8.4.tar.gz#sha256=51a1a934e7ed0ff221bdd91bf9d3b604d875afbb3aa2367133503fee168f5bfa (from -r requirements.txt (line 37))
  Using cached virtualenvwrapper-4.8.4.tar.gz (334 kB)
Collecting virtualenv-clone==0.5.4
  Using cached virtualenv_clone-0.5.4-py2.py3-none-any.whl (6.6 kB)
Collecting qtconsole==4.7.4
  Using cached qtconsole-4.7.4-py2.py3-none-any.whl (118 kB)
Collecting pyzmq==19.0.1
  Using cached pyzmq-19.0.1-cp38-cp38-manylinux1_x86_64.whl (1.1 MB)
Collecting QtPy==1.9.0
  Using cached QtPy-1.9.0-py2.py3-none-any.whl (54 kB)
Collecting jupyter-client==6.1.3
  Using cached jupyter_client-6.1.3-py3-none-any.whl (106 kB)
Collecting https://files.pythonhosted.org/packages/95/84/119a46d494f008969bf0c775cb2c6b3579d3c4cc1bb1b41a022aa93ee242/tornado-6.0.4.tar.gz#sha256=0fe2d45ba43b00a41cd73f8be321a44936dc1aba233dee979f17a042b83eb6dc (from jupyter-client==6.1.3->jupyter-console==6.1.0->jupyter==1.0.0->-r requirements.txt (line 38))
  Using cached tornado-6.0.4.tar.gz (496 kB)
Collecting python-dateutil==2.8.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting Pygments==2.6.1
  Using cached Pygments-2.6.1-py3-none-any.whl (914 kB)
Collecting Django==1.11.29
  Using cached Django-1.11.29-py2.py3-none-any.whl (6.9 MB)
Collecting bleach==3.1.5
  Using cached bleach-3.1.5-py2.py3-none-any.whl (151 kB)
Collecting webencodings==0.5.1
  Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting packaging==20.4
  Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting pylint-django==2.0.15
  Using cached pylint_django-2.0.15-py3-none-any.whl (70 kB)
Collecting pylint-plugin-utils==0.6
  Using cached pylint_plugin_utils-0.6-py3-none-any.whl (10 kB)
Collecting pylint==2.4.4
  Using cached pylint-2.4.4-py3-none-any.whl (302 kB)
Collecting mccabe==0.6.1
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting astroid==2.3.3
  Using cached astroid-2.3.3-py3-none-any.whl (205 kB)
Collecting https://files.pythonhosted.org/packages/23/84/323c2415280bc4fc880ac5050dddfb3c8062c2552b34c2e512eb4aa68f79/wrapt-1.11.2.tar.gz#sha256=565a021fd19419476b9362b05eeaa094178de64f8361e44468f9e9d7843901e1 (from astroid==2.3.3->pylint==2.4.4->-r requirements.txt (line 35))
  Using cached wrapt-1.11.2.tar.gz (27 kB)
Collecting lazy-object-proxy==1.4.3
  Using cached lazy_object_proxy-1.4.3-cp38-cp38-manylinux1_x86_64.whl (58 kB)
Collecting isort==4.3.21
  Using cached isort-4.3.21-py2.py3-none-any.whl (42 kB)
Collecting ipykernel==5.3.0
  Using cached ipykernel-5.3.0-py3-none-any.whl (119 kB)
Collecting https://files.pythonhosted.org/packages/10/dc/a4933487ea8322336d3827c1193fff0ddd8b3a43eeda3f446f163dacc407/pudb-2019.2.tar.gz#sha256=e8f0ea01b134d802872184b05bffc82af29a1eb2f9374a277434b932d68f58dc (from -r requirements.txt (line 39))
  Using cached pudb-2019.2.tar.gz (59 kB)
Collecting https://files.pythonhosted.org/packages/45/dd/d57924f77b0914f8a61c81222647888fbb583f89168a376ffeb5613b02a6/urwid-2.1.0.tar.gz#sha256=0896f36060beb6bf3801cb554303fef336a79661401797551ba106d23ab4cd86 (from pudb==2019.2->-r requirements.txt (line 39))
  Using cached urwid-2.1.0.tar.gz (630 kB)
Collecting ipywidgets==7.5.1
  Using cached ipywidgets-7.5.1-py2.py3-none-any.whl (121 kB)
Collecting widgetsnbextension==3.5.1
  Using cached widgetsnbextension-3.5.1-py2.py3-none-any.whl (2.2 MB)
Collecting notebook==6.0.3
  Using cached notebook-6.0.3-py3-none-any.whl (9.7 MB)
Collecting terminado==0.8.3
  Using cached terminado-0.8.3-py2.py3-none-any.whl (33 kB)
Collecting ptyprocess==0.6.0
  Using cached ptyprocess-0.6.0-py2.py3-none-any.whl (39 kB)
Collecting Send2Trash==1.5.0
  Using cached Send2Trash-1.5.0-py3-none-any.whl (12 kB)
Collecting https://files.pythonhosted.org/packages/b3/23/41a5a24b502d35a4ad50a5bb7202a5e1d9a0364d0c12f56db3dbf7aca76d/prometheus_client-0.7.1.tar.gz#sha256=71cd24a2b3eb335cb800c7159f423df1bd4dcd5171b234be15e3f31ec9f622da (from notebook==6.0.3->jupyter==1.0.0->-r requirements.txt (line 38))
  Using cached prometheus_client-0.7.1.tar.gz (38 kB)
Collecting pyparsing==2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting numpy==1.18.4
  Using cached numpy-1.18.4-cp38-cp38-manylinux1_x86_64.whl (20.7 MB)
Collecting ipython==7.14.0
  Downloading ipython-7.14.0-py3-none-any.whl (782 kB)
     |████████████████████████████████| 782 kB 3.0 MB/s 
Collecting https://files.pythonhosted.org/packages/1b/b3/0708228132a772de8802967af71d00c6b1f2f407e6f8efcc7055ac5900fe/backcall-0.1.0.zip#sha256=bbbf4b1e5cd2bdb08f915895b51081c041bac22394fdfcfdfbe9f14b77c08bf2 (from ipython==7.14.0->jupyter-console==6.1.0->jupyter==1.0.0->-r requirements.txt (line 38))
  Downloading backcall-0.1.0.zip (11 kB)
Collecting pickleshare==0.7.5
  Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting pexpect==4.8.0
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting jedi==0.17.0
  Downloading jedi-0.17.0-py2.py3-none-any.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 4.7 MB/s 
Collecting parso==0.7.0
  Downloading parso-0.7.0-py2.py3-none-any.whl (100 kB)
     |████████████████████████████████| 100 kB 7.6 MB/s 
Collecting pandas==1.0.3
  Using cached pandas-1.0.3-cp38-cp38-manylinux1_x86_64.whl (10.0 MB)
Collecting pytz==2020.1
  Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting stevedore==1.32.0
  Using cached stevedore-1.32.0-py2.py3-none-any.whl (43 kB)
Collecting pbr==5.4.5
  Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting https://files.pythonhosted.org/packages/ba/af/ee46390c14f270c891ecc5bd4cd3f02c7be408bb2d36bd96b1ae92ccb544/PyNLPl-1.2.9.tar.gz#sha256=a7f9db301fe03ac3c98b115a3d000e52cfff46fad10d8df849708a0838a548e5 (from -r requirements.txt (line 16))
  Using cached PyNLPl-1.2.9.tar.gz (277 kB)
Collecting rdflib==5.0.0
  Using cached rdflib-5.0.0-py3-none-any.whl (231 kB)
Collecting isodate==0.6.0
  Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting httplib2==0.18.1
  Using cached httplib2-0.18.1-py3-none-any.whl (95 kB)
Collecting lxml==4.5.1
  Using cached lxml-4.5.1-cp38-cp38-manylinux1_x86_64.whl (5.4 MB)
Collecting gunicorn==20.0.4
  Using cached gunicorn-20.0.4-py2.py3-none-any.whl (77 kB)
Collecting boto==2.49.0
  Using cached boto-2.49.0-py2.py3-none-any.whl (1.4 MB)
Collecting cmd2==1.0.2
  Using cached cmd2-1.0.2-py3-none-any.whl (111 kB)
Collecting https://files.pythonhosted.org/packages/f6/5b/55866e1cde0f86f5eec59dab5de8a66628cb0d53da74b8dbc15ad8dabda3/pyperclip-1.8.0.tar.gz#sha256=b75b975160428d84608c26edba2dec146e7799566aea42c1fe1b32e72b6028f2 (from cmd2==1.0.2->-r requirements.txt (line 28))
  Using cached pyperclip-1.8.0.tar.gz (16 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/home/abeka/.pyenv/versions/3.8.2/envs/temp/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/home/abeka/.pyenv/versions/3.8.2/envs/temp/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/home/abeka/.pyenv/versions/3.8.2/envs/temp/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 332, in run
    requirement_set = resolver.resolve(
  File "/home/abeka/.pyenv/versions/3.8.2/envs/temp/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 80, in resolve
    self._result = resolver.resolve(requirements)
  File "/home/abeka/.pyenv/versions/3.8.2/envs/temp/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 413, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/abeka/.pyenv/versions/3.8.2/envs/temp/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 323, in resolve
    raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 100

I tried to play with this a little, and when I remove jupyter it succeeds in resolving and installing everything. I also tried to resolve this requirement file using pip-compile from pip-tools, which succeeded, so it is possible to install all of these together.

Thanks for the awesome work,
Beka

[amitbeka]

@nlhkabu @ei8fdb

[deveshks]

There has recently been a fix the set the number of rounds to a large number (2000000) before ResolutionTooDeep hits in #8275.

I also believe that this fix has been added in the beta version released today (pip 20.2b1) which can be used by doing pip install -U pip --pre which upgrades pip to the beta version.

[pfmoore]

I also believe that this fix has been added in the beta version released today (pip 20.2b1) which can be used by doing pip install -U pip --pre which upgrades pip to the beta version.

Correct. @amitbeka Please can you re-try with the new beta and see if that fixes the issue for you? And thanks for taking the time to report this, it's much appreciated.

[amitbeka]

I just tried the dev version (cloned the git repo) before seeing your message, and it works! closing :)

[pfmoore]

Excellent news, thanks again!