Extra constraints are added during locking

[jxltom]

Issue description

It happens for version from 2018.6.25. Version 2018.5.18 and older don't have this issue.

Following pipfile can not be locked. From the setup.py, django-elasticsearch-dsl = "==0.5.0" requires elasticsearch-dsl>=2.1.0,<7.0.0', and elasticsearch-dsl = "==5.0.0" requires 'elasticsearch>=5.0.0,<6.0.0'. There should be no problem with locking.

But pipenv add elasticsearch<7.0.0,>=6.0.0 in the second round of locking, which is obviously wrong.

There is something wrong with https://github.com/pypa/pipenv/blob/master/pipenv/patched/notpip/_internal/resolve.py. I tried to fix it, but the content of requirement_set in each run is not deterministic which make it hard to debug.

[packages]
django-elasticsearch-dsl = "==0.5.0"
elasticsearch-dsl = "==5.0.0"

Expected result

pipenv lock --clear should work.

Actual result

pipenv lock --clear doesn't work and there is the verbose logs.

$ pipenv lock --clear --verbose

Creating a virtualenv for this project…
Pipfile: /tmp/pipenv-ul2fe_70-project/Pipfile
Using /home/jxltom/.local/share/virtualenvs/pipenv-AxtSETHD/bin/python3.6m (3.6.6) to create virtualenv…
Already using interpreter /home/jxltom/.local/share/virtualenvs/pipenv-AxtSETHD/bin/python3.6m
Using real prefix '/home/jxltom/.pyenv/versions/3.6.6'
New python executable in /tmp/pipenv-ul2fe_70-project/.venv/bin/python3.6m
Also creating executable in /tmp/pipenv-ul2fe_70-project/.venv/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /tmp/pipenv-ul2fe_70-project/.venv
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
using sources: [{'url': 'https://pypi.org/simple', 'verify_ssl': True, 'name': 'pypi'}]
Using pip: -i https://pypi.org/simple

                          ROUND 1
Current constraints:
  django-elasticsearch-dsl==0.5.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-l1p1ryex-constraints.txt (line 2))
  elasticsearch-dsl==5.0.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-l1p1ryex-constraints.txt (line 3))

Finding the best candidates:
  found candidate django-elasticsearch-dsl==0.5.0 (constraint was ==0.5.0)
  found candidate elasticsearch-dsl==5.0.0 (constraint was ==5.0.0)

Finding secondary dependencies:
  django-elasticsearch-dsl==0.5.0 not in cache, need to check index
  django-elasticsearch-dsl==0.5.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4" requires django-elasticsearch-dsl==0.5.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4", elasticsearch-dsl<7.0.0,>=2.1.0, elasticsearch<7.0.0,>=6.0.0, ipaddress, python-dateutil, six, urllib3>=1.21.1
  elasticsearch-dsl==5.0.0 not in cache, need to check index
  elasticsearch-dsl==5.0.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4" requires elasticsearch-dsl==5.0.0; python_version >= "2.6"
and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4", elasticsearch<6.0.0,>=5.0.0, python-dateutil, six, urllib3>=1.21.1

New dependencies found in this round:
  adding ['django-elasticsearch-dsl', '==0.5.0', '[]']
  adding ['elasticsearch', '<6.0.0,<7.0.0,>=5.0.0,>=6.0.0', '[]']
  adding ['elasticsearch-dsl', '<7.0.0,==5.0.0,>=2.1.0', '[]']
  adding ['ipaddress', '', '[]']
  adding ['python-dateutil', '', '[]']
  adding ['six', '', '[]']
  adding ['urllib3', '>=1.21.1', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  django-elasticsearch-dsl==0.5.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-l1p1ryex-constraints.txt (line 2))
  elasticsearch<6.0.0,<7.0.0,>=5.0.0,>=6.0.0
  elasticsearch-dsl<7.0.0,==5.0.0,>=2.1.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-l1p1ryex-constraints.txt (line 3))
  ipaddress
  python-dateutil
  six
  urllib3>=1.21.1

Finding the best candidates:
  found candidate django-elasticsearch-dsl==0.5.0 (constraint was ==0.5.0)
Using pip: -i https://pypi.org/simple

                          ROUND 1
Current constraints:
  django-elasticsearch-dsl==0.5.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-27yb5vni-constraints.txt (line 2))
  elasticsearch-dsl==5.0.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-27yb5vni-constraints.txt (line 3))

Finding the best candidates:
  found candidate django-elasticsearch-dsl==0.5.0 (constraint was ==0.5.0)
  found candidate elasticsearch-dsl==5.0.0 (constraint was ==5.0.0)

Finding secondary dependencies:
  django-elasticsearch-dsl==0.5.0 not in cache, need to check index
  django-elasticsearch-dsl==0.5.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4" requires django-elasticsearch-dsl==0.5.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4", elasticsearch-dsl<7.0.0,>=2.1.0, elasticsearch<7.0.0,>=6.0.0, ipaddress, python-dateutil, six, urllib3>=1.21.1
  elasticsearch-dsl==5.0.0 not in cache, need to check index
  elasticsearch-dsl==5.0.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4" requires elasticsearch-dsl==5.0.0; python_version >= "2.6"
and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4", elasticsearch<6.0.0,>=5.0.0, python-dateutil, six, urllib3>=1.21.1

New dependencies found in this round:
  adding ['django-elasticsearch-dsl', '==0.5.0', '[]']
  adding ['elasticsearch', '<6.0.0,<7.0.0,>=5.0.0,>=6.0.0', '[]']
  adding ['elasticsearch-dsl', '<7.0.0,==5.0.0,>=2.1.0', '[]']
  adding ['ipaddress', '', '[]']
  adding ['python-dateutil', '', '[]']
  adding ['six', '', '[]']
  adding ['urllib3', '>=1.21.1', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  django-elasticsearch-dsl==0.5.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-27yb5vni-constraints.txt (line 2))
  elasticsearch<6.0.0,<7.0.0,>=5.0.0,>=6.0.0
  elasticsearch-dsl<7.0.0,==5.0.0,>=2.1.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-27yb5vni-constraints.txt (line 3))
  ipaddress
  python-dateutil
  six
  urllib3>=1.21.1

Finding the best candidates:
  found candidate django-elasticsearch-dsl==0.5.0 (constraint was ==0.5.0)

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches elasticsearch<6.0.0,<7.0.0,>=5.0.0,>=6.0.0
Tried: 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.4.4, 0.4.5, 0.4.5, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.9.0, 1.9.0, 2.0.0, 2.0.0, 2.1.0, 2.1.0, 2.2.0, 2.2.0, 2.3.0, 2.3.0, 2.4.0, 2.4.0, 2.4.1, 2.4.1, 5.0.0, 5.0.0, 5.0.1, 5.0.1, 5.1.0, 5.1.0, 5.2.0, 5.2.0, 5.3.0, 5.3.0, 5.4.0, 5.4.0, 5.5.0, 5.5.0, 5.5.1, 5.5.1, 5.5.2, 5.5.2, 5.5.3, 5.5.3, 6.0.0, 6.0.0, 6.1.1, 6.1.1, 6.2.0, 6.2.0, 6.3.0, 6.3.0, 6.3.1, 6.3.1
There are incompatible versions in the resolved dependencies.
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches elasticsearch<6.0.0,<7.0.0,>=5.0.0,>=6.0.0
Tried: 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.4.4, 0.4.5, 0.4.5, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.9.0, 1.9.0, 2.0.0, 2.0.0, 2.1.0, 2.1.0, 2.2.0, 2.2.0, 2.3.0, 2.3.0, 2.4.0, 2.4.0, 2.4.1, 2.4.1, 5.0.0, 5.0.0, 5.0.1, 5.0.1, 5.1.0, 5.1.0, 5.2.0, 5.2.0, 5.3.0, 5.3.0, 5.4.0, 5.4.0, 5.5.0, 5.5.0, 5.5.1, 5.5.1, 5.5.2, 5.5.2, 5.5.3, 5.5.3, 6.0.0, 6.0.0, 6.1.1, 6.1.1, 6.2.0, 6.2.0, 6.3.0, 6.3.0, 6.3.1, 6.3.1
There are incompatible versions in the resolved dependencies.

---

[techalchemy]

This is a super annoying artifact related to the dependency resolution implementation (which isn't implemented directly in pip, incidentally, so you can't fix it there)

I have a fix for this which I already implemented upstream in pip-tools, which is responsible for the resolution of this -- but the root cause is basically that we are using resolve instead of _resolve_one on the resolution set which, starting in pip 10, has the unfortunate consequence of feeding the originating requirement back into the dependency set which means we do a round of resolution and then we attempt another round, this time with effectively two competing copies of the same dependency which may be in conflict or resolve non-deterministically.

It would be a super long shot to hope anyone would find that, it took roughly everyone involved in maintaining pip-tools to sort that out -- not to mention we also patch pip-tools like crazy in pipenv.

As far as the issue, it's likely a clone of the many other resolution issues out there and is probably fixed by the PR I just put up :)

[jxltom]

Thanks a lot for your explanation.

Awesome PR, let's see what happens after this PR is merged. 🎉 🎉 🎉

[jxltom]

I'm too impatient to wait it being merged. I just tested it in my local forked branch, it works like a charm, at least for this issue.

I will close this as soon as #2935 is merged. @techalchemy Thank you for the PR! 🤝

[jxltom]

closes by #2935