Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extra constraints are added during locking #2933

Closed
jxltom opened this issue Oct 6, 2018 · 4 comments
Closed

Extra constraints are added during locking #2933

jxltom opened this issue Oct 6, 2018 · 4 comments

Comments

@jxltom
Copy link
Contributor

@jxltom jxltom commented Oct 6, 2018

Issue description

It happens for version from 2018.6.25. Version 2018.5.18 and older don't have this issue.

Following pipfile can not be locked. From the setup.py, django-elasticsearch-dsl = "==0.5.0" requires elasticsearch-dsl>=2.1.0,<7.0.0', and elasticsearch-dsl = "==5.0.0" requires 'elasticsearch>=5.0.0,<6.0.0'. There should be no problem with locking.

But pipenv add elasticsearch<7.0.0,>=6.0.0 in the second round of locking, which is obviously wrong.

There is something wrong with https://github.com/pypa/pipenv/blob/master/pipenv/patched/notpip/_internal/resolve.py. I tried to fix it, but the content of requirement_set in each run is not deterministic which make it hard to debug.

[packages]
django-elasticsearch-dsl = "==0.5.0"
elasticsearch-dsl = "==5.0.0"

Expected result

pipenv lock --clear should work.

Actual result

pipenv lock --clear doesn't work and there is the verbose logs.

$ pipenv lock --clear --verbose

Creating a virtualenv for this project…
Pipfile: /tmp/pipenv-ul2fe_70-project/Pipfile
Using /home/jxltom/.local/share/virtualenvs/pipenv-AxtSETHD/bin/python3.6m (3.6.6) to create virtualenv…
Already using interpreter /home/jxltom/.local/share/virtualenvs/pipenv-AxtSETHD/bin/python3.6m
Using real prefix '/home/jxltom/.pyenv/versions/3.6.6'
New python executable in /tmp/pipenv-ul2fe_70-project/.venv/bin/python3.6m
Also creating executable in /tmp/pipenv-ul2fe_70-project/.venv/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /tmp/pipenv-ul2fe_70-project/.venv
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
using sources: [{'url': 'https://pypi.org/simple', 'verify_ssl': True, 'name': 'pypi'}]
Using pip: -i https://pypi.org/simple

                          ROUND 1
Current constraints:
  django-elasticsearch-dsl==0.5.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-l1p1ryex-constraints.txt (line 2))
  elasticsearch-dsl==5.0.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-l1p1ryex-constraints.txt (line 3))

Finding the best candidates:
  found candidate django-elasticsearch-dsl==0.5.0 (constraint was ==0.5.0)
  found candidate elasticsearch-dsl==5.0.0 (constraint was ==5.0.0)

Finding secondary dependencies:
  django-elasticsearch-dsl==0.5.0 not in cache, need to check index
  django-elasticsearch-dsl==0.5.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4" requires django-elasticsearch-dsl==0.5.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4", elasticsearch-dsl<7.0.0,>=2.1.0, elasticsearch<7.0.0,>=6.0.0, ipaddress, python-dateutil, six, urllib3>=1.21.1
  elasticsearch-dsl==5.0.0 not in cache, need to check index
  elasticsearch-dsl==5.0.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4" requires elasticsearch-dsl==5.0.0; python_version >= "2.6"
and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4", elasticsearch<6.0.0,>=5.0.0, python-dateutil, six, urllib3>=1.21.1

New dependencies found in this round:
  adding ['django-elasticsearch-dsl', '==0.5.0', '[]']
  adding ['elasticsearch', '<6.0.0,<7.0.0,>=5.0.0,>=6.0.0', '[]']
  adding ['elasticsearch-dsl', '<7.0.0,==5.0.0,>=2.1.0', '[]']
  adding ['ipaddress', '', '[]']
  adding ['python-dateutil', '', '[]']
  adding ['six', '', '[]']
  adding ['urllib3', '>=1.21.1', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  django-elasticsearch-dsl==0.5.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-l1p1ryex-constraints.txt (line 2))
  elasticsearch<6.0.0,<7.0.0,>=5.0.0,>=6.0.0
  elasticsearch-dsl<7.0.0,==5.0.0,>=2.1.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-l1p1ryex-constraints.txt (line 3))
  ipaddress
  python-dateutil
  six
  urllib3>=1.21.1

Finding the best candidates:
  found candidate django-elasticsearch-dsl==0.5.0 (constraint was ==0.5.0)
Using pip: -i https://pypi.org/simple

                          ROUND 1
Current constraints:
  django-elasticsearch-dsl==0.5.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-27yb5vni-constraints.txt (line 2))
  elasticsearch-dsl==5.0.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-27yb5vni-constraints.txt (line 3))

Finding the best candidates:
  found candidate django-elasticsearch-dsl==0.5.0 (constraint was ==0.5.0)
  found candidate elasticsearch-dsl==5.0.0 (constraint was ==5.0.0)

Finding secondary dependencies:
  django-elasticsearch-dsl==0.5.0 not in cache, need to check index
  django-elasticsearch-dsl==0.5.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4" requires django-elasticsearch-dsl==0.5.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4", elasticsearch-dsl<7.0.0,>=2.1.0, elasticsearch<7.0.0,>=6.0.0, ipaddress, python-dateutil, six, urllib3>=1.21.1
  elasticsearch-dsl==5.0.0 not in cache, need to check index
  elasticsearch-dsl==5.0.0; python_version >= "2.6" and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4" requires elasticsearch-dsl==5.0.0; python_version >= "2.6"
and python_version != "3.1.*" and python_version != "3.2.*" and python_version != "3.3.*" and python_version != "3.0.*" and python_version < "4", elasticsearch<6.0.0,>=5.0.0, python-dateutil, six, urllib3>=1.21.1

New dependencies found in this round:
  adding ['django-elasticsearch-dsl', '==0.5.0', '[]']
  adding ['elasticsearch', '<6.0.0,<7.0.0,>=5.0.0,>=6.0.0', '[]']
  adding ['elasticsearch-dsl', '<7.0.0,==5.0.0,>=2.1.0', '[]']
  adding ['ipaddress', '', '[]']
  adding ['python-dateutil', '', '[]']
  adding ['six', '', '[]']
  adding ['urllib3', '>=1.21.1', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  django-elasticsearch-dsl==0.5.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-27yb5vni-constraints.txt (line 2))
  elasticsearch<6.0.0,<7.0.0,>=5.0.0,>=6.0.0
  elasticsearch-dsl<7.0.0,==5.0.0,>=2.1.0 (from -r /tmp/pipenv-w2knmten-requirements/pipenv-27yb5vni-constraints.txt (line 3))
  ipaddress
  python-dateutil
  six
  urllib3>=1.21.1

Finding the best candidates:
  found candidate django-elasticsearch-dsl==0.5.0 (constraint was ==0.5.0)

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches elasticsearch<6.0.0,<7.0.0,>=5.0.0,>=6.0.0
Tried: 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.4.4, 0.4.5, 0.4.5, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.9.0, 1.9.0, 2.0.0, 2.0.0, 2.1.0, 2.1.0, 2.2.0, 2.2.0, 2.3.0, 2.3.0, 2.4.0, 2.4.0, 2.4.1, 2.4.1, 5.0.0, 5.0.0, 5.0.1, 5.0.1, 5.1.0, 5.1.0, 5.2.0, 5.2.0, 5.3.0, 5.3.0, 5.4.0, 5.4.0, 5.5.0, 5.5.0, 5.5.1, 5.5.1, 5.5.2, 5.5.2, 5.5.3, 5.5.3, 6.0.0, 6.0.0, 6.1.1, 6.1.1, 6.2.0, 6.2.0, 6.3.0, 6.3.0, 6.3.1, 6.3.1
There are incompatible versions in the resolved dependencies.
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches elasticsearch<6.0.0,<7.0.0,>=5.0.0,>=6.0.0
Tried: 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.4.4, 0.4.5, 0.4.5, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.9.0, 1.9.0, 2.0.0, 2.0.0, 2.1.0, 2.1.0, 2.2.0, 2.2.0, 2.3.0, 2.3.0, 2.4.0, 2.4.0, 2.4.1, 2.4.1, 5.0.0, 5.0.0, 5.0.1, 5.0.1, 5.1.0, 5.1.0, 5.2.0, 5.2.0, 5.3.0, 5.3.0, 5.4.0, 5.4.0, 5.5.0, 5.5.0, 5.5.1, 5.5.1, 5.5.2, 5.5.2, 5.5.3, 5.5.3, 6.0.0, 6.0.0, 6.1.1, 6.1.1, 6.2.0, 6.2.0, 6.3.0, 6.3.0, 6.3.1, 6.3.1
There are incompatible versions in the resolved dependencies.

@techalchemy
Copy link
Member

@techalchemy techalchemy commented Oct 7, 2018

This is a super annoying artifact related to the dependency resolution implementation (which isn't implemented directly in pip, incidentally, so you can't fix it there)

I have a fix for this which I already implemented upstream in pip-tools, which is responsible for the resolution of this -- but the root cause is basically that we are using resolve instead of _resolve_one on the resolution set which, starting in pip 10, has the unfortunate consequence of feeding the originating requirement back into the dependency set which means we do a round of resolution and then we attempt another round, this time with effectively two competing copies of the same dependency which may be in conflict or resolve non-deterministically.

It would be a super long shot to hope anyone would find that, it took roughly everyone involved in maintaining pip-tools to sort that out -- not to mention we also patch pip-tools like crazy in pipenv.

As far as the issue, it's likely a clone of the many other resolution issues out there and is probably fixed by the PR I just put up :)

Loading

@jxltom
Copy link
Contributor Author

@jxltom jxltom commented Oct 7, 2018

Thanks a lot for your explanation.

Awesome PR, let's see what happens after this PR is merged. 🎉 🎉 🎉

Loading

@jxltom
Copy link
Contributor Author

@jxltom jxltom commented Oct 7, 2018

I'm too impatient to wait it being merged. I just tested it in my local forked branch, it works like a charm, at least for this issue.

I will close this as soon as #2935 is merged. @techalchemy Thank you for the PR! 🤝

Loading

@jxltom
Copy link
Contributor Author

@jxltom jxltom commented Oct 8, 2018

closes by #2935

Loading

@jxltom jxltom closed this Oct 8, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants