After upgrading patroni to 1.4.4 getting error ContextualVersionConflict on starting patroni

[debraj-manna]

I am trying to upgrade patroni to 1.4.4 from 1.3.6. After upgrade and starting patroni I was seeing the below error

Traceback (most recent call last):
  File "/usr/local/bin/patroni", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3037, in <module>
    @_call_aside
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3021, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3050, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 657, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 670, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 860, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('urllib3<1.23,>=1.21.1'), set(['requests']))

To get around this I have to to the following

sudo pip uninstall urllib3
sudo pip install urllib3==1.22

Is this expected?

[CyberDem0n]

Hmm, that's interesting.
Requirement urllib3<1.23,>=1.21.1 is not coming from Patroni directly...

[debraj-manna]

@CyberDem0n - Just to addurllib3 is released yesterday as shown here -
https://pypi.org/project/urllib3/#history
It looks something similar as mentioned in the second part of #694 .

[CyberDem0n]

This is not something that we have an influence on.

One day ago urllib3==1.23 was released, which is required not only by patroni, but also by requests module, and the last one has very strict version check: 'urllib3>=1.21.1,<1.23'.

The only possible solution would be to require absolutely the same versions of urllib3 as requests does, but even this won't work reliably, because urllib3 is required by a bunch of other modules and some of them definitely don't have a strict constraint on upper version boundary (python-etcd and kubernetes are examples of such modules).

The only we can do is to wait for a new release of requests.

[hjacobs]

@CyberDem0n that does not sound like an satisfying answer, do we have a broken Docker image and can't do anything about it? I would not close this bug in that case..

[debraj-manna]

@hjacobs @CyberDem0n - I cannot reopen this. If you feel this needs to be reopened, you can.

[c4milo]

I'm running into this issue as well. This is an issue in Patroni for not locking dependency versions properly. Please assess https://github.com/pypa/pipenv, it is very helpful for this purpose.

[hjacobs]

I'm reopening this issue as new comments from @debraj-manna and @c4milo indicate that the issue is not resolved yet.

[CyberDem0n]

As I already explained, urllib3==1.23 was released about a week ago. It is used in requests module, but the requests explicitly requires urllib3<=1.22.

From our side, I don't see any single reason to limit maximum versions for neither requests, nor urllib3, because if we go this way, we will have to update dependencies every single time when the new module is released.

We can't live with very old versions either, because our goal is to be able to run Patroni with the least possible amount of external dependencies. Ideally all dependencies should be installed from packages of your favorite distro. The latest Debian and Ubuntu actually allow this. Moreover, they provide Patroni package!

And actually talking about weird dependency system in python. I am really surprised that nobody yet complained about psycopg2:

>>> import psycopg2
/home/akukushkin/git/patroni/vpy31/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.

The funny thing is, psycopg2-binary and psycopg2 are two different modules, but they install files in the same place on filesystem! You can even install both at the same time and pip won't complain, but overwrite existing files. If you decide to remove one of the above modules, the second one will stop working, because files are deleted...

Anyway, requests==2.19.0 was released 6 hours ago. That means the problem is solved.

[c4milo]

@CyberDem0n, FWIW, breakages accidentally introduced by upstream dependencies will keep happening unless dependencies are properly locked. It's a widely accepted best practice to lock down dependencies. Example: https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/