New resolver silently resolves conflicts differently from the previous resolver

[joestringer]

What did you want to do?

At a high level, build the documentation for the Cilium project. We observed that automated builds began to fail around or after 30 November 2020.

More specifically, use pip to manage dependencies where we have a package that is provided differently depending on the architecture.

Input

requirements.txt:

...
git+git://github.com/cilium/sphinx_rtd_theme.git@v0.7; platform_machine != "aarch64"
...
sphinx-rtd-theme==0.2.4
...

Usage

pip install -r requirements.txt

On earlier pip versions, when running on non-aarch64 systems, this would silently resolve the conflict to install the forked version of the package from git. On aarch64 systems, the package would be provided by the upstream sphinx-rtd-theme package.

Output

After upgrading to pip 20.3, the conflict was silently resolved in the alternate order which subsequently lead to build problems due to the dependency being resolved differently.

Additional information

We use pip for this purpose in two ways:

• For local development, we have a Docker container based on Alpine Linux which appears to pull relatively new versions of pip relatively frequently.
• For build of the the documentation, we rely on readthedocs.org. Readthedocs appears to automatically upgrade pip to the latest version from pypi for every single build.

Workarounds:

• Downgrade Alpine Linux to an earlier release that doesn't pull the latest pip
• Specify --use-deprecated=legacy-resolver on the pip commandline

For readthedocs, neither of these options are available as we are not in control of the pip commandline, only the requirements.txt file.

Having looked at the issue for a while now, I believe that ultimately the cause of this issue stems from the ambiguity in the requirements.txt file, so by clarifying the conflict in the requirements file, we were able to mitigate the issue (cilium/cilium#14264). We were not previously aware that the requirements.txt was ambiguous, or it did not seem to matter as the end result (build the docs successfully) was achieved.

Expected behavior from pip

I would have preferred if pip rejected my combination of requirements, informing me that there is a conflict, and providing me with a message that tells me that this is the problem and I can either resolve it (ideally giving some hints as to what the problem is, at least which packages are involved in the conflict) or alternatively, specify some override flag to tell pip "I don't care, figure it out whichever way you want and I will deal with the consequences" (which would have the same behaviour as pip 20.3 today).

At this point I tried to figure out if there are any requirements.txt linters or --strict-mode or anything that I could enable to detect such problems and inform me in advance, but so far I was unable to locate any such tools.

[uranusjr]

The algorithms (both new and old) guarentee to return a solution that meets all user requests, but never promises it would be “the” result. In practice, the result returned by either resolver is deterministic, otherwise it would be extremely difficult to debug them otherwise, but there is no guarantee it would always match exactly what you want, unless you tell it what you want.

In other words, this is expected, and I don’t plan to change it because the behaviour is IMO correct. If you need specific versions of packages, you should tell pip about it.

[brainwane]

Hello @joestringer and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us.

You may find a few useful troubleshooting and workaround tips from the documentation. I'm also interested in your point

At this point I tried to figure out if there are any requirements.txt linters or --strict-mode or anything that I could enable to detect such problems and inform me in advance, but so far I was unable to locate any such tools.

It's tough that you cannot run pip check (since you don't control the environment that the pip commands are run in, just the requirements.txt file). Is this something that RtD is thinking of addressing?

[joestringer]

I tried running pip check locally after installing the requirements, and it did not notice the incompatibility:

$ pip check
No broken requirements found.

I assume this is because the resolution of the requirements file is inconsistent, it's not the already-installed libraries that are inconsistent after processing the requirements file. The requirements file itself has a conflict, but pip randomly and silently chooses one selection of the conflicting packages to install and this successfully puts some set of packages in place.

[joestringer]

At this point I think it's probably a feature request to detect conflicting packages as specified in the requirements.txt file, but from a technical perspective I honestly don't have a solid idea of how this could even be determined so I'm happy to close out this issue.