Entering backtracking loop in github3 vs. Flask App Builder #10924

potiuk · 2022-02-25T17:58:04Z

Description

The lattest version of pip 22.0.3 (but also 21.* line) enters the backtracking loop that goes on for hours when there are (it seems) three conflicting versions of dependencies in specific case for Airflow.

It took me about a day to figure out where the problems comes from as it was entirely not obvious and I had to do trial and error (and develop some tools to help) so it would be great to handle it better. Maybe this report wil help to improve the backtracking solution.

The problem is that suddenly (~ 2 days ago) Airflow eager upgrade builds when we try to upgrade dependencies to latest versions, started to time out after more than 80 minutes (this is the CI timeout we have for our builds).

The problem turned out to be triggered by github3.py dependency. This dependency released a 3.1.2 version which had PyJWT>=2.3.0. The same limits are set in (released few ago 3.1.0, 3.1.1 version of github3.py). Another depoendency of ours - Flask App Builder has PyJWT<2.0.0.

Until 3.1.2 version has been released for Gtihub.py, everything worked fine. There is a github3.py dependency (3.0.0) which does not depend at all on PyJWT and previously resolving the dependencies for Airlfow correctly resolved to this version. However when Github3.py 3.1.2 was released, pip started to enter a long backtracking loop running for hours (normally such installation is < 10 minutes).

I investigated and tracked it down to one simple change that switches between backtracting <> no backtracking.

Adding limitation github3.py<3.1.2 leads to pip correctly resolve the dependencies (and use github3.py == 3.0.0)
Switching back to github3.py leads to backtracking and pip unable to find that github3.py 3.0.0 is a good candidate.

It almost looks like backtracking for other dependencies kicks when a dependency has 3 conflicting latest versions.

Unfortunately we do not yet have a solution to be able to see the root cause of the problem (discussed in #10258) so tracking of the problem was terribly painful and inolved finding which dependencies were upgraded in certain days and trying to limit them manually. This is an extremely painful process - we try to improve it in airflow by adding some workarounds and mostly recording of snapshots of dependencies in the CI, byt a solution in pip that could flag at least potential culprits in connection with built-in time limits for resolution could help with such investigations even if it is impossible to find a perfect resolution. Currently when such a bactrack happens, it is like finding a needle in a haystack.

Expected behavior

I think there are two options:

solving this case properly - there is a solution, but apparently current backtracking algorithm cannot find it. I do not know enough details about the algorithm, but maybe it can be improved to handle this case (however I do realise it might be complex or impossible)
adding timeout on resolution for CI case (and ctrl-c for manual running ) and providing information about potentially conflicting dependencies would be second best solution. See also discussion in Present found conflicts when trying to resolve them #10258 where this has been discussed.

Even if we cannot find the solution but the user gets some hints on what causes it, this would be of great help for the maintainers and users of complex dependency sets. The problem in this case might be (like in our case) that some - completely random dependency gets released and starts impossible-to-resolve backtracking without the knowledge or influence from the package maintainer. This is quite bad, because one day your package will install, but the second day it will stop installing. But at least detecting and flagging the potential problems might help the maintainers of those packages to at least understand what the root cause of the problem is.

pip version

22.0.3 (and 21.* line too).

Python version

3.8 (also)

OS

Debian Buster (docker image).

How to Reproduce

Run:

docker run -it ghcr.io/apache/airflow/main/ci/python3.7:72588ef409f416d6341928c278bf7ed86939a133

You will enter the Python 3.7 Airflow CI image that has tje "golden" set of dependencies (from today) of Airlfow.

Run eager upgrade command:

pip install ".[devel_all]" --upgrade --upgrade-strategy eager "dill<0.3.3" "certifi<2021.0.0" "google-ads<14.0.1"

(Note the other limtis are hard-coded beofre in order to overcome similar eager behaviour or pip behaviour where install_requires conflics with airflow extras)

You will see the pip command enter a very, very long backtracking.

Run eager upgrade command with additional limitation on github3.py:

pip install ".[devel_all]" --upgrade --upgrade-strategy eager "dill<0.3.3" "certifi<2021.0.0" "google-ads<14.0.1" "github3.py<3.1.2"

This one will properly install airflow with github3 version == 3.0.0.

When you run pipdeptree | less and look for PyJWT, you will find that PyJWT<2.0 is limited by Flask Application Builder.
The installed version of github3.py is 3.0.0 (as expected).

Apparently (this is a wild guess though) having just two conflicting versions of github3.py to consider (3.1.1 and 3.1.0) does not enter backtracking loop, but having 3 (3.1.0, 3.1.1, 3.1.2) enter the backtracking loop. All three 3.1.* github3.py have PyJWT >=2.3.0. The 3.0.0 has no PyJWT limitation at all.

Output

Output of the "backtracking" (stopped after several minutes of backtracking) is here:

https://gist.github.com/potiuk/3cf30ddf546ebaf6c80f3352e73fa8f9

Backtracking continues after this log for at least 70 minutes (I was not patient enough to wait) withoug making any progress.

Output for successfull installation (with "github3.py<3.1.2") is here:

https://gist.github.com/potiuk/2691ef71b0a124ff6b74f5653ee01d81

This takes less than 5 minutes on my machine.

Code of Conduct

I agree to follow the PSF Code of Conduct.

The text was updated successfully, but these errors were encountered:

Github3 version 3.1.2 requires PyJWT>=2.3.0 which clashes with Flask App Builder where PyJWT is <2.0.0 Actually GitHub3.1.0 already introduced PyJWT>=2.3.0 but so far `pip` was able to resolve it without getting into a long backtracking loop and figure out that github3 3.0.0 version is the right version similarly limiting it to 3.1.2 causes pip not to enter the backtracking loop. Apparently when there Are 3 versions with PyJWT>=2.3.0 (3.1.0, 3.1.1 an 3.1.2) pip enters into backtrack loop and fails to resolve that github3 3.0.0 is the right version to use. This limitation could be removed if PyJWT limitation < 2.0.0 is dropped from FAB or when pip resolution is improved to handle the case, The issue which describes this PIP behaviour and hopefully allowing to improve it is tracked in pypa/pip#10924.

* Limits GitHub3.py in order to avoid backtracking Github3 version 3.1.2 requires PyJWT>=2.3.0 which clashes with Flask App Builder where PyJWT is <2.0.0 Actually GitHub3.1.0 already introduced PyJWT>=2.3.0 but so far `pip` was able to resolve it without getting into a long backtracking loop and figure out that github3 3.0.0 version is the right version similarly limiting it to 3.1.2 causes pip not to enter the backtracking loop. Apparently when there Are 3 versions with PyJWT>=2.3.0 (3.1.0, 3.1.1 an 3.1.2) pip enters into backtrack loop and fails to resolve that github3 3.0.0 is the right version to use. This limitation could be removed if PyJWT limitation < 2.0.0 is dropped from FAB or when pip resolution is improved to handle the case, The issue which describes this PIP behaviour and hopefully allowing to improve it is tracked in pypa/pip#10924.

This is a follow-up after investigation done with failing main builds (resulting in apache#21824 and tracked in pypa/pip#10924)

This is a follow-up after investigation done with failing main builds (resulting in apache#21824 and tracked in pypa/pip#10924) Handling failure of image building has been also improved as part of the PR. Instead of separate "cancel" job (which did not really work anyway) we build and push empty images instead and the empty images are handled in the "wait for images" job with appropriate message.

…21825) * Add CI jobs and tooling to aid with tracking backtracking pip issues This is a follow-up after investigation done with failing main builds (resulting in #21824 and tracked in pypa/pip#10924) Handling failure of image building has been also improved as part of the PR. Instead of separate "cancel" job (which did not really work anyway) we build and push empty images instead and the empty images are handled in the "wait for images" job with appropriate message. * Update dev/breeze/src/airflow_ci/find_newer_dependencies.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

* Limits GitHub3.py in order to avoid backtracking Github3 version 3.1.2 requires PyJWT>=2.3.0 which clashes with Flask App Builder where PyJWT is <2.0.0 Actually GitHub3.1.0 already introduced PyJWT>=2.3.0 but so far `pip` was able to resolve it without getting into a long backtracking loop and figure out that github3 3.0.0 version is the right version similarly limiting it to 3.1.2 causes pip not to enter the backtracking loop. Apparently when there Are 3 versions with PyJWT>=2.3.0 (3.1.0, 3.1.1 an 3.1.2) pip enters into backtrack loop and fails to resolve that github3 3.0.0 is the right version to use. This limitation could be removed if PyJWT limitation < 2.0.0 is dropped from FAB or when pip resolution is improved to handle the case, The issue which describes this PIP behaviour and hopefully allowing to improve it is tracked in pypa/pip#10924. GitOrigin-RevId: 16f5185d7fa01df9432bb1228dcd676c35745222

…(#21825) * Add CI jobs and tooling to aid with tracking backtracking pip issues This is a follow-up after investigation done with failing main builds (resulting in #21824 and tracked in pypa/pip#10924) Handling failure of image building has been also improved as part of the PR. Instead of separate "cancel" job (which did not really work anyway) we build and push empty images instead and the empty images are handled in the "wait for images" job with appropriate message. * Update dev/breeze/src/airflow_ci/find_newer_dependencies.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> GitOrigin-RevId: 0b9257b870bc5fbf724a0c818316f6cb79043355