pip 20.2b1 resolver still failing with ResolutionTooDeep for jupyter install

[billgale]

Description

Building jupyter exceeds the incredibly large number of 2,000,000 (Set max_rounds to an incredibly large number #8275) for the pip 20.2b1 dependency resolver recommended in (pip resolver fails in a fresh env when installing jupyter with other stuff #8290).

Docker Image: docker:18.09.7 / docker:18.09.7-dind
Pip: pip 20.2b1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
Python: Python 3.8.5

Expected behavior

It would nice if we didn't have 2 million dependencies evaluated. I am all for an improved dependency resolver, but this is not an improvement for this case. I had to roll back to juypterhub 1.0.0 to circumvent this.

pip version

20.2b1

Python version

3.8.5

OS

Docker Image: docker:18.09.7 / docker:18.09.7-dind

How to Reproduce

Dockerfile

FROM jupyterhub/jupyterhub:1.4.0                                                              

# python packages
COPY config/pip.conf /root/.pip/
RUN python3 -m pip --version
#RUN python3 -m pip --no-cache-dir install --upgrade pip
# ensure pip depenency resolver does not fail with ResolutionTooDeep
# ref: https://github.com/pypa/pip/issues/8290
RUN python3 -m pip --no-cache-dir install --upgrade pip==20.2b1
RUN python3 -m pip --version
RUN python3 --version
COPY config/requirements.txt /tmp/
RUN python3 -m pip install --no-cache-dir -r /tmp/requirements.txt

requirements.txt

tpsubmitter==0.52.0                                                             
tpscripts==0.34.0
awscli
jupyterlab
jupyterhub-ldapauthenticator
jupyter_contrib_nbextensions
psycopg2-binary
pandas==0.25.0
SQLAlchemy==1.3.6

Output

INFO: pip is looking at multiple versions of jupyter-packaging to determine which version is compatible with other requirements. This could take a while.
841  Downloading jupyter_packaging-0.8.0-py2.py3-none-any.whl (13 kB)
842  Downloading jupyter_packaging-0.7.12-py2.py3-none-any.whl (10 kB)
843  Downloading jupyter_packaging-0.7.11-py2.py3-none-any.whl (10 kB)
844  Downloading jupyter_packaging-0.7.10-py2.py3-none-any.whl (10 kB)
845  Downloading jupyter_packaging-0.7.9-py2.py3-none-any.whl (10 kB)
846INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
847  Downloading jupyter_packaging-0.7.8-py2.py3-none-any.whl (10 kB)
848  Downloading jupyter_packaging-0.7.7-py2.py3-none-any.whl (10 kB)
849  Downloading jupyter_packaging-0.7.6-py2.py3-none-any.whl (10 kB)
850  Downloading jupyter_packaging-0.7.5-py2.py3-none-any.whl (10 kB)
851  Downloading jupyter_packaging-0.7.4-py2.py3-none-any.whl (10 kB)
852  Downloading jupyter_packaging-0.7.3-py2.py3-none-any.whl (10 kB)
853  Downloading jupyter_packaging-0.7.1-py2.py3-none-any.whl (10 kB)
854  Downloading jupyter_packaging-0.7.0-py2.py3-none-any.whl (10 kB)
855INFO: pip is looking at multiple versions of jinja2 to determine which version is compatible with other requirements. This could take a while.
856Collecting jinja2>=2.10
857  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
858  Downloading Jinja2-2.11.1-py2.py3-none-any.whl (126 kB)
859  Downloading Jinja2-2.11.0-py2.py3-none-any.whl (126 kB)
860  Downloading Jinja2-2.10.3-py2.py3-none-any.whl (125 kB)
861  Downloading Jinja2-2.10.2-py2.py3-none-any.whl (125 kB)
862  Downloading Jinja2-2.10.1-py2.py3-none-any.whl (124 kB)
863  Downloading Jinja2-2.10-py2.py3-none-any.whl (126 kB)
864INFO: pip is looking at multiple versions of jinja2 to determine which version is compatible with other requirements. This could take a while.
865ERROR: Exception:
866Traceback (most recent call last):
867  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/base_command.py", line 180, in _main
868    status = self.run(options, args)
869  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/req_command.py", line 204, in wrapper
870    return func(self, options, args)
871  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/commands/install.py", line 318, in run
872    requirement_set = resolver.resolve(
873  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/resolver.py", line 127, in resolve
874    result = self._result = resolver.resolve(
875  File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 473, in resolve
876    state = resolution.resolve(requirements, max_rounds=max_rounds)
877  File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 384, in resolve
878    raise ResolutionTooDeep(max_rounds)
879pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000
880The command '/bin/sh -c python3 -m pip install --no-cache-dir -r /tmp/requirements.txt' returned a non-zero code: 2
`

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

There have been many improvements since 20.2b1. Is this reproducible with the latest pip version (21.1.1)?

[billgale]

Correct hit the same issue in 21.1.1 and worked back to 20.2.b1 to see if this was a solution, but hit the same limit.

In the end solved the problem by backing off docker image's jupytherhub to 1.0.0.

[notatallshaw]

Do you still have this issue in pip 21.3.1? A significant backtracking optimization landed in pip 21.3, and the ecosystem has largely improved.

I couldn't reproduce your example exactly as it seems to depend on internal packages I don't have access to.

[billgale]

@notatallshaw The packages "tpsubmitter" and "tpscripts" are internal, and I do not believe necessary to reproduce this error. Can you try striking them, and changing the docker image to "jupyterhub:1.4.0"? If that does not produce the error, let me know.

I am not doing python development now so hoping you can reproduce with this modification. If not let me know and I will have to recreate my python development env and work the issue along with you.

[notatallshaw]

@notatallshaw The packages "tpsubmitter" and "tpscripts" are internal, and I do not believe necessary to reproduce this error. Can you try striking them, and changing the docker image to "jupyterhub:1.4.0"? If that does not produce the error, let me know. Appreciate your looking into this after so long after the issue was reported.

Your OP already uses "jupyterhub:1.4.0" right?

Following the instructions as is I get an error related to building pandas. I was able to fix this by adding these lines before installing the requirements file:

RUN apt-get update && apt install build-essential python3-dev -y
RUN python3 -m pip install cython numpy==1.18

After that the docker file built fine.

[notatallshaw]

This should now be resolved on Pip main (28239f9) due to sarugaku/resolvelib#111 and sarugaku/resolvelib#113. I haven't been able to reproduce any other long backtracking issue with these improvements.

Can you retry by installing Pip like so: python -m pip install git+https://github.com/pypa/pip.git?

[notatallshaw]

You can now install Pip 23.1 which has massively improved backtracking capability, I was never able to reproduce your issue but if you have clear instructions that didn't involved internal packages I would be happy to try.

[pradyunsg]

Given that we haven't reproduced this in over an year and with the recent improvements, I'll close this out.