Adding an option to allow pip-compile to accept conflicts

[twig]

Hi there,

I'm trying out pip-tools in hope of cleaning up my requirements.txt file and quickly found an issue while trying out some examples.

It seems that two levels of version pinning on the same library causes pip-tools to conflict with itself.

Environment Versions

1. OS Type: Ubuntu 14
2. Python version: 2.7.6
3. pip version: 9.0.1
4. pip-tools version: 1.9.0

Steps to replicate

1. virtualenv testpiptools
2. pip install --upgrade pip
3. source testpiptools/bin/activate
4. pip install pip-tools
5. edit requests.in and edit it

vero==2.0.0

6. pip-compile requirements.in
7. requirements.txt contains

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
requests==2.7.0           # via vero
vero==2.0.0

(this is fine so far)
8. add latest version of requests to requirements.in

vero==2.0.0
requests==2.18.4

9 . error occurs (see below)

Expected result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
requests==2.18.4           # via vero
vero==2.0.0
# whatever else is required by requests at v2.18.4

Actual result

Using indexes:
  https://pypi.python.org/simple

                          ROUND 1
Current constraints:
  requests==2.18.4
  vero==2.0.0

Finding the best candidates:
  found candidate requests==2.18.4 (constraint was ==2.18.4)
  found candidate vero==2.0.0 (constraint was ==2.0.0)

Finding secondary dependencies:
  vero==2.0.0               requires requests==2.7.0
  requests==2.18.4          requires certifi>=2017.4.17, chardet<3.1.0,>=3.0.2, idna<2.7,>=2.5, urllib3<1.23,>=1.21.1

New dependencies found in this round:
  adding [u'certifi', '>=2017.4.17', '[]']
  adding [u'chardet', '<3.1.0,>=3.0.2', '[]']
  adding [u'idna', '<2.7,>=2.5', '[]']
  adding [u'requests', '==2.7.0', '[]']
  adding [u'urllib3', '<1.23,>=1.21.1', '[]']
Removed dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  certifi>=2017.4.17
  chardet<3.1.0,>=3.0.2
  idna<2.7,>=2.5
  requests==2.18.4,==2.7.0
  urllib3<1.23,>=1.21.1
  vero==2.0.0

Finding the best candidates:
  found candidate certifi==2017.7.27.1 (constraint was >=2017.4.17)
  found candidate chardet==3.0.4 (constraint was >=3.0.2,<3.1.0)
  found candidate idna==2.6 (constraint was >=2.5,<2.7)
Could not find a version that matches requests==2.18.4,==2.7.0
Tried: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.4.0, 0.4.1, 0.5.0, 0.5.1, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.8.7, 0.8.8, 0.8.9, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.6, 0.10.7, 0.10.8, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.13.5, 0.13.6, 0.13.7, 0.13.8, 0.13.9, 0.14.0, 0.14.1, 0.14.2, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 2.0.0, 2.0.0, 2.0.1, 2.0.1, 2.1.0, 2.1.0, 2.2.0, 2.2.0, 2.2.1, 2.2.1, 2.3.0, 2.3.0, 2.4.0, 2.4.0, 2.4.1, 2.4.1, 2.4.2, 2.4.2, 2.4.3, 2.4.3, 2.5.0, 2.5.0, 2.5.1, 2.5.1, 2.5.2, 2.5.2, 2.5.3, 2.5.3, 2.6.0, 2.6.0, 2.6.1, 2.6.1, 2.6.2, 2.6.2, 2.7.0, 2.7.0, 2.8.0, 2.8.0, 2.8.1, 2.8.1, 2.9.0, 2.9.0, 2.9.1, 2.9.1, 2.9.2, 2.9.2, 2.10.0, 2.10.0, 2.11.0, 2.11.0, 2.11.1, 2.11.1, 2.12.0, 2.12.0, 2.12.1, 2.12.1, 2.12.2, 2.12.2, 2.12.3, 2.12.3, 2.12.4, 2.12.4, 2.12.5, 2.12.5, 2.13.0, 2.13.0, 2.14.0, 2.14.0, 2.14.1, 2.14.1, 2.14.2, 2.14.2, 2.15.1, 2.15.1, 2.16.0, 2.16.0, 2.16.1, 2.16.1, 2.16.2, 2.16.2, 2.16.3, 2.16.3, 2.16.4, 2.16.4, 2.16.5, 2.16.5, 2.17.0, 2.17.0, 2.17.1, 2.17.1, 2.17.2, 2.17.2, 2.17.3, 2.17.3, 2.18.0, 2.18.0, 2.18.1, 2.18.1, 2.18.2, 2.18.2, 2.18.3, 2.18.3, 2.18.4, 2.18.4

so the problem seems to be requests==2.18.4 cannot also be ==2.7.0 at the same time.

I've read through the documentation on the github page many times and even tried:

pip-compile -v requirements.in -P requests==2.18.4

Many blogs refer to this as a great tool and I have been itching to try it, but this seems like a rather common issue to run into fairly early.

Is there something missing from a FAQ of which I haven't seen yet?

[vphilippon]

Hi @twig,

This is actually a normal behaviour: pip-compile's job is to find candidates that respects all constraints.
In this specific case, you required requests==2.18.4, and vero requires specifically requests==2.7.0.
You effectively end up with a version constraints conflict, and that's what pip-compile is telling in the end.

By installing requests==2.18.4, it would go against the requirements of vero. If you were to do a straight pip install -r requirements.in followed by a pip check, it would tell you that the requirements of vero are not met, having requests==2.18.4 while requests==2.7.0 is required.

I hope that makes sense.
We likely need an explanation of this in the README, you're not the first one to ask.

[twig]

Wow you responded much quicker than expected.
And yes, I think having that in the docs would be very helpful.
Thanks for that.

Feels rather frustrating that there isn't an override of some sort though as bringing more libraries over to requests.in would result in even more varieties of version mismatches, especially for something as popular as the requests library.

[vphilippon]

You might be interested to know that vero v2.0.1 has unpinned requests to avoid unnecessary dependencies conflicts.
Related issue: waveaccounting/vero-python/issues/26

Now, about that override option, let me present it this way: by overriding the version of requests installed, you would deliberately go and install a version that goes against the expectations of some of your other dependencies that explicitly stated it required requests==2.7.0 exactly.

There are cases where it could be ok, but in most cases, it comes down to having packages defining the right requirements constraints for their use cases instead. A lib strictly pinning some of its dependencies should have a really good reason for doing so.

[twig]

Ahh yes, I came across that when playing with pip-tools earlier. I chose it because v2.0.0 was such a simple example to reproduce.

Hmm I agree with you on the explicit versioning for a reason, but for libraries which have been abandoned or are very slow moving (pyrax for example), we don't really have a choice because using a fork of a repo doesn't seem to work as it would in the regular requirements.txt file (as I've read in #272)

[twig]

Actually, scrap that. I just realised our old requirements.txt was using a different URL format which pip-compile doesn't like https://github.com/USERNAME/REPO/archive/COMMITHASH.zip#egg=PACKAGENAME==VERSION. Converting it to -e git://github.com/USERNAME/REPO.git@COMMITHASH#egg=PACKAGENAME==VERSION works a treat!

It would probably be worth noting in the docs as well I suppose.

[jcerjak]

Explicitly pinning versions in a library is obviously not desired. But the world is dark and full of terrors: either a library is not well maintained, or it needs to pin a dependency to a specific version due to monkey-patching, or any other exotic or not-so-exotic reason.

This is probably why for the above example:

# requirements.in 
vero==2.0.0
requests==2.18.4

pip (tested with version 10.0.0) will only print a warning but still finish the installation:

$ pip install -r requirements.in
...
vero 2.0.0 has requirement requests==2.7.0, but you'll have requests 2.18.4 which is incompatible.

while pip-tools will raise an error:

$ pip-compile
...
Could not find a version that matches requests==2.18.4,==2.7.0

I agree that raising an error by default makes sense, I'd prefer if pip would do that as well. But IMHO there should be an option in pip-tools to only print a warning. If an application developer has pinned all the dependencies, explicitly enabled the option to not treat these warnings as errors and accepted the risk, then we should allow him/her to do so.

[jcerjak]

Here is a quick proof-of-concept how this could work.

Example requirements.in:

vero==2.0.0
requests==2.18.4

Default behavior (unchanged):

$ pip-compile
Could not find a version that matches requests==2.18.4,==2.7.0
...
There are incompatible versions in the resolved dependencies.

With the --allow-conflicts option:

$ pip-compile --allow-conflicts
Conflicting versions found: requests==2.18.4,==2.7.0, using our constraints: requests==2.18.4
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
--index-url https://pypi.org/simple

certifi==2018.4.16        # via requests
chardet==3.0.4            # via requests
idna==2.6                 # via requests
requests==2.18.4
urllib3==1.22             # via requests
vero==2.0.0

Would be nice if we included a comment in requirements.txt, that requests had conflicts, but this would require a bit more changes.

Thoughts?

[vphilippon]

First @jcerjak, thanks for the PoC. If I get this right, the idea in your PoC is that when getting a conflict, with --allow-conflicts, we try again with the original requirement from the requirements.in file as an override, and if that requirement can be met, we warn the user and go on.

The idea makes some sense as it's similar to the way pip goes through requirements by default: handling only the top-level or first met dependency requirement. If I was to agree on adding this option (oops, spoilers!), that would likely be the way I would suggest to do it.

But, here are my concerns. First, the technical concerns:

• Make sure that this not corrupt the dependency cache in some way that registers that requests==2.18.4 is a valid dependency for vero==2.0.0.
• Make sure that re-running pip-compile without --allow-conflicts and an existing requirements.txt will not keep requests==2.18.4 and will raise a conflict error again.
• Check what happens with pip-sync with a file like that. In this case, I'm pretty sure that the pip install vero==2.0.0 done by pip-sync will uninstall requests==2.18.4 and install requests==2.7.0, as it processes the lines one-by-one.

Beyond that, my main concern is this: This option allows users to easily create a requirements.txt file that is incoherent, and break one of the promise of pip-tools. Why use pip-tools then? Just use pip install+pip freeze > requirements.txt. Maybe use pip check to spot some incompatibility and fix them as needed. As I've mentioned, the suggested behavior is like pip: If someone doesn't need the guarantee of coherency and reproducibility from pip-tools, why use pip-tools? IMO, this guaranteed coherency is one of pip-tools's strength.

pipenv has taken a similar approach, the --skip-lock option, which falls back to installing as pip would, essentially skipping the pip-compile part. I believe this is sane: do not create a Lockfile (or requirements.txt) that is incoherent.

I believe those dependencies conflicts should be addressed at the lib level, rather than making it easier to ignore the problems. I believe that lib maintainers should be encouraged to be thoughtful on their dependencies constraints, and adopt good practice that are future-proof (ex: leave the pinning and downpining to the app level, give leeway). If a lib is slow-moving, then I would highly suggest that the maintainers avoid adding tight constraints that will be a pain to users. For the abandoned lib, that's a shame, and that falls in the list of consequences of un-maintained libs.

So, in my opinion, this would hurt more than it would help down the line. I do not see this as an harmless option that will help some users, I see this as opposed to what pip-tools tries to achieve.

Now, this is my take on this. This is an open-source and Jazzband project. Anyone who disagrees with my vision is strongly encouraged to say so. If this is a widely desired option, then OK, I'm not here to impose my vision on everyone. I'm acting as Lead, not Boss.

[jcerjak]

Thanks @vphilippon for a timely and thorough response. I do agree that ideally this should be addressed at the lib level, though sometimes you want to have an override switch to get things done. But I understand that adding this would add more complexity and maintenance burden and is also not in line with the concept of pip-tools.

So I'll probably go with the approach of creating a fresh virtualenv + pip freeze for now. Thanks!

[twig]

Why use pip-tools then? Just use pip install+pip freeze > requirements.txt

Actually, I'm mainly pip-tools to help keep reference of why libraries are in my requirements.txt in the first place.

The auto-generated file includes comments which are very helpful in debugging why libraries are there, something which regular pip freeze doesn't do.

The ability to override when needed would still be highly appreciated!

[jcerjak]

Actually, I'm mainly pip-tools to help keep reference of why libraries are in my requirements.txt in the first place.

Not as nice, but you can manually check why a library was installed with pipdeptree:

$ pip install pip-tools pipdeptree

Check dependencies:

$ pipdeptree
pip-tools==2.0.1
  - click [required: >=6, installed: 6.7]
  - first [required: Any, installed: 2.0.1]
  - six [required: Any, installed: 1.11.0]
pipdeptree==0.12.1
  - pip [required: >=6.0.0, installed: 10.0.1]
setuptools==39.0.1
wheel==0.31.0

Check which packages require click and pip:

$ pipdeptree --reverse --packages click,pip
click==6.7
  - pip-tools==2.0.1 [requires: click>=6]
pip==10.0.1
  - pipdeptree==0.12.1 [requires: pip>=6.0.0]

[twig]

Neat! Sadly not as nice as having it in requirements.txt as pip-tools would have it

Just an update, my initial issue was resolved by the library authors at waveaccounting/vero-python#26

But it doesn't really resolve the issue in the case of inactive libraries and such.

From memory, npm/yarn has a nice resolution selector when it runs into version conflicts.

[jdufresne]

This request looks like a duplicate of #215.

[jace]

I've found a related problem via pip-compile-multi, with libraries being unnecessarily restrictive with dependencies. In my case:

requirements/base.in:
	sqlalchemy[asyncio]
	# Resolves to require greenlet==3.0.1, the latest release (no pin specified in SQLAlchemy)
requirements/test.in:
	playwright
	# Resolves to require greenlet==2.0.2 because that version is pinned (unnecessarily)

This causes pip-compile-multi to error out with: Package greenlet was resolved to different versions in different environments: 2.0.2 and 3.0.1

Playwright works fine with Greenlet 3.0.1 as it has no API changes, and I need that version for Python 3.12 support, but I'm now stuck waiting for Playwright to remove that pin or at least to update it. Even if I forgo 3.12 support, I'll have to pin greenlet in my dependencies — despite it not being a direct dependency — just to get compilation to work:

requirements/base.in:
	greenlet<3.0
	sqlalchemy[asyncio]
requirements/test.in:
	playwright

But how do I know when playwright has removed that pin? If they update it, I'll get a helpful compilation error, but if they remove the pin, I won't know unless I'm specifically reading their release notes or watching relevant GitHub issues.

awscli is similarly unnecessarily restrictive with supported versions, and in their case I had to drop the dependency to get other dependencies to work. It'll really help to ignore restrictions from specific libraries like this, trusting my test cases to catch actual compatibility problems.