New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipfile python_version not taken into account when generating lockfile #1901

Closed
rainyday opened this Issue Apr 3, 2018 · 21 comments

Comments

Projects
None yet
4 participants
@rainyday

rainyday commented Apr 3, 2018

Pipenv lock/install/sync generates a Pipfile.lock with package versions incompatible with the version of Python

output of python -m pipenv.help: https://pastebin.com/raw/r4WCG5AR


Expected result

Pipenv generates a lockfile with installable packages and then installs them.

Actual result

Pipenv generates a Pipfile.lock containing requirements that are incompatible with the version of Python specified in Pipfile using python_version or python_full_version and installation fails. Example:

Error message:

Collecting pylint-quotes==0.1.9 (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-_auxxp4x-requirements/pipenv-u4w5lmul-requirement.txt (line 1))
  1 location(s) to search for versions of pylint-quotes:
  * https://pypi.python.org/simple/pylint-quotes/
  Getting page https://pypi.python.org/simple/pylint-quotes/
  Looking up "https://pypi.python.org/simple/pylint-quotes/" in the cache
  Current age based on date: 11
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  The response is "fresh", returning cached response
  600 > 11
  Analyzing links from page https://pypi.python.org/simple/pylint-quotes/
    Found link https://pypi.python.org/packages/7c/88/7a894a269a58a907fe173282f50d642dcbaa44a0b5d12043f4785347f11b/pylint-quotes-0.1.3.tar.gz#md5=2351ca090248406e0eeafa75d7a1eb10 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.3
    Found link https://pypi.python.org/packages/a0/0a/372a8219a6a42a9fdba7bdc00aaffb3a9d88d01138022d76c08f6b83764a/pylint_quotes-0.1.3-py2.py3-none-any.whl#md5=1097eeffc8f0a75cc843cc0c97866be7 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.3
    Found link https://pypi.python.org/packages/7d/a7/b1ddde1d38c1141ab2d7c51d307ef7f81d62b970fc035ea8db74911821c4/pylint-quotes-0.1.5.tar.gz#md5=1cd3ebbcfc518653c81593d2486da0d0 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.5
    Found link https://pypi.python.org/packages/18/69/f19c51efaf3951fb570a78243b261a7bf178e66468f3692e1cd225fb4cfb/pylint_quotes-0.1.5-py2.py3-none-any.whl#md5=5cc78a567e4c25aa300c4bf933819e8b (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.5
    Found link https://pypi.python.org/packages/8c/c4/04c56b0446962f6c7424d76d5799747d40ad3472684e8995ba01a3fcdbdc/pylint-quotes-0.1.6.tar.gz#md5=86f79ae93e06a6fc488c7f73924be812 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.6
    Found link https://pypi.python.org/packages/e7/ad/b3037ce9414ab986052d395b1b9cb6026e6e296afade18b2840d1363a073/pylint_quotes-0.1.6-py2.py3-none-any.whl#md5=1213fed2daec3507c9207d356de72b06 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.6
    Found link https://pypi.python.org/packages/8d/ef/a135a29c4c4358b60712ef44b2eac21dda876ab9d0203bb4f254800d4255/pylint-quotes-0.1.7.tar.gz#md5=389d3ec970ef4b8d42d8247df3d08373 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.7
    Found link https://pypi.python.org/packages/e8/b5/f03009ab72fe1d1442c4b8c3fc06680ecb4c1f0385b5b0bf2bf221b3195e/pylint_quotes-0.1.7-py2.py3-none-any.whl#md5=dea6d20e390fe80013d8a33be932d206 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.7
    Found link https://pypi.python.org/packages/1c/b9/e14468554607a685abcb0d617d1ef7c5f25e5cbb1e280fed370594993006/pylint-quotes-0.1.8.tar.gz#md5=7330dcdb88e941ad2f03048ba95ede05 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.8
    Found link https://pypi.python.org/packages/28/f5/e00a9b7597ba7b92b58ddac472f9cf5bfd347022a65a4beafd5f682a7758/pylint_quotes-0.1.8-py2.py3-none-any.whl#md5=fb713d376b27abdbd9491b126e85b5a5 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.8
    The package https://pypi.python.org/packages/ba/20/4e59a01fd90fbe67385b790cb6ddb4a462f4d8e15548bab27c59c3b23fd3/pylint-quotes-0.1.9.tar.gz#md5=02eaa4853463f8f867f36e031b2503f0 (from https://pypi.python.org/simple/pylint-quotes/) (requires-python:>=3.4) is incompatible with the pythonversion in use. Acceptable python versions are:>=3.4
    The package https://pypi.python.org/packages/c2/cc/afc270fa43a12d10736422ddaa653c7c7664561f10b182b76884438637c2/pylint_quotes-0.1.9-py2.py3-none-any.whl#md5=7c680717a5af1c59e68809c865478f82 (from https://pypi.python.org/simple/pylint-quotes/) (requires-python:>=3.4) is incompatible with the pythonversion in use. Acceptable python versions are:>=3.4
  Could not find a version that satisfies the requirement pylint-quotes==0.1.9 (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-_auxxp4x-requirements/pipenv-u4w5lmul-requirement.txt (line 1)) (from versions: 0.1.3, 0.1.5, 0.1.6, 0.1.7, 0.1.8)
Cleaning up...
No matching distribution found for pylint-quotes==0.1.9 (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-_auxxp4x-requirements/pipenv-u4w5lmul-requirement.txt (line 1))
Exception information:
Traceback (most recent call last):
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/index.py", line 514, in find_requirement
    'No matching distribution found for %s' % req
DistributionNotFound: No matching distribution found for pylint-quotes==0.1.9 (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-_auxxp4x-requirements/pipenv-u4w5lmul-requirement.txt (line 1))

An error occurred while installing pylint-quotes==0.1.9! Will try again.
Steps to replicate

Create a Pipfile that specifies a python version and packages with a version range that includes versions that are both compatible and incompatible with the python_version specified.

Example Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[requires]
python_version = "2.7"
python_full_version = "2.7.14"

[packages]
beaker = "==1.9.0"
bottle = "==0.12.13"
funcsigs = "==1.0.2"
gevent = "==1.2.2"
greenlet = "==0.4.12"
influxdb = "<=6.0.0"
jsoncomment = "==0.3.0"
netaddr = "==0.7.19"
numpy = "==1.11.2"
pandas = "==0.19.1"
pyrad = "==2.1"
python-dateutil = "==2.6.1"
pytz = "==2017.2"
requests = "*"
schedule = "==0.4.2"
tacacs-plus = "==2.0"
ujson = "==1.35"
bson = "*"
pymongo = "*"
"h5py" = "*"
environs = "*"
pyyaml = "*"
deprecated = "*"
appdirs = "*"
pycryptodome = "==3.4.11"
click = "*"
pyopenssl = "*"
pem = "*"

[dev-packages]
pytest = "*"
rope = "*"
ipython = "*"
mock = "*"
pylint = "*"
pylint-quotes = "*"
yapf = "*"
teamcity-messages = "*"
webtest = "*"
requests-mock = "*"
pdbpp = "*"
pytest-mock = "*"
pytest-cov = "*"
pytest-pythonpath = "*"

With this Pipfile both pylint-quotes and ipython will fail since pylint-quotes 1.9 and ipython 6.3.0 are both incompatible with python 2.7.14

@rainyday rainyday changed the title from Pipenv python_version into account when generating lockfile to Pipfile python_version not taken into account when generating lockfile Apr 3, 2018

@techalchemy

This comment has been minimized.

Member

techalchemy commented Apr 4, 2018

Thanks for the report, sorry for the issue :( this is a know issue and is fixed in master, closing for now. Will have a release out soon

@techalchemy techalchemy closed this Apr 4, 2018

@frostming

This comment has been minimized.

Collaborator

frostming commented Apr 13, 2018

This issue does exist on pipenv 11.10.0. The root cause is the resolver doesn't compare the current python version with python_requires and always picks the latest one.

Pipfile content:

[requires]
python_version = "2.7"

[packages]
django = "*"    # python2.7 requires django < 2
$ python -V
Python 2.7.14
pipenv lock verbose output
Locking [packages] dependencies…

Using pip: -i https://pypi.python.org/simple

                          ROUND 1                           
Current constraints:
  django

Finding the best candidates:
  found candidate django==2.0.4 (constraint was <any>)

Finding secondary dependencies:
  django==2.0.4 not in cache, need to check index
  django==2.0.4             requires pytz

New dependencies found in this round:
  adding [u'pytz', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  django
  pytz

Finding the best candidates:
  found candidate django==2.0.4 (constraint was <any>)
  found candidate pytz==2018.4 (constraint was <any>)

Finding secondary dependencies:
  django==2.0.4             requires pytz
  pytz==2018.4 not in cache, need to check index
  pytz==2018.4              requires -
------------------------------------------------------------
Result of round 2: stable, done


Updated Pipfile.lock (0454a4)!
@rainyday

This comment has been minimized.

rainyday commented Apr 18, 2018

This is not fixed

@techalchemy

This comment has been minimized.

Member

techalchemy commented Apr 18, 2018

To clarify, the words in your pipfile are used to generate your environment; the environment itself performs resolution. I need to see the output of pipenv lock —verbose to really understand what’s going on but I’m guessing based on the small part that you included that it one of your dependencies actually requires this version

@rainyday

This comment has been minimized.

rainyday commented Apr 18, 2018

Here is a simplified Pipfile that reproduces the issue

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
ipython = "*"

[dev-packages]

[requires]
python_version = "2.7"
$ pipenv lock -vvv
Creating a virtualenv for this project…
Using /Users/<user>/.pyenv/versions/2.7.14/bin/python2.7 (2.7.14) to create virtualenv…
⠋Running virtualenv with interpreter /Users/<user>/.pyenv/versions/2.7.14/bin/python2.7
New python executable in /Users/<user>/.local/share/virtualenvs/pipenv-test-3vVgCXNF/bin/python2.7
Also creating executable in /Users/<user>/.local/share/virtualenvs/pipenv-test-3vVgCXNF/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/<user>/.local/share/virtualenvs/pipenv-test-3vVgCXNF
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:
  ipython

Finding the best candidates:
  found candidate ipython==6.3.1 (constraint was <any>)

Finding secondary dependencies:
  ipython==6.3.1            requires appnope; sys_platform == "darwin", backcall, decorator, jedi>=0.10, pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2, typing; python_version <= "3.4"

New dependencies found in this round:
  adding [u'appnope', '', '[]']
  adding [u'backcall', '', '[]']
  adding [u'decorator', '', '[]']
  adding [u'jedi', '>=0.10', '[]']
  adding [u'pexpect', '', '[]']
  adding [u'pickleshare', '', '[]']
  adding [u'prompt-toolkit', '<2.0.0,>=1.0.15', '[]']
  adding [u'pygments', '', '[]']
  adding [u'simplegeneric', '>0.8', '[]']
  adding [u'traitlets', '>=4.2', '[]']
  adding [u'typing', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  appnope
  backcall
  decorator
  ipython
  jedi>=0.10
  pexpect
  pickleshare
  prompt-toolkit<2.0.0,>=1.0.15
  pygments
  simplegeneric>0.8
  traitlets>=4.2
  typing

Finding the best candidates:
  found candidate appnope==0.1.0 (constraint was <any>)
  found candidate backcall==0.1.0 (constraint was <any>)
  found candidate decorator==4.3.0 (constraint was <any>)
  found candidate ipython==6.3.1 (constraint was <any>)
  found candidate jedi==0.12.0 (constraint was >=0.10)
  found candidate pexpect==4.5.0 (constraint was <any>)
  found candidate pickleshare==0.7.4 (constraint was <any>)
  found candidate prompt-toolkit==1.0.15 (constraint was >=1.0.15,<2.0.0)
  found candidate pygments==2.2.0 (constraint was <any>)
  found candidate simplegeneric==0.8.1 (constraint was >0.8)
  found candidate traitlets==4.3.2 (constraint was >=4.2)
  found candidate typing==3.6.4 (constraint was <any>)

Finding secondary dependencies:
  pygments==2.2.0           requires -
  prompt-toolkit==1.0.15    requires six>=1.9.0, wcwidth
  decorator==4.3.0          requires -
  jedi==0.12.0              requires parso>=0.2.0
  backcall==0.1.0           requires -
  traitlets==4.3.2          requires decorator, enum34; python_version == "2.7", ipython-genutils, six
  pickleshare==0.7.4        requires pathlib2; python_version in "2.6 2.7 3.2 3.3"
  ipython==6.3.1            requires appnope; sys_platform == "darwin", backcall, decorator, jedi>=0.10, pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2, typing; python_version <= "3.4"
  simplegeneric==0.8.1      requires -

New dependencies found in this round:
  adding [u'enum34', '', '[]']
  adding [u'ipython-genutils', '', '[]']
  adding [u'parso', '>=0.2.0', '[]']
  adding [u'pathlib2', '', '[]']
  adding [u'ptyprocess', '>=0.5', '[]']
  adding [u'six', '>=1.9.0', '[]']
  adding [u'wcwidth', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 2: not stable

                          ROUND 3
Current constraints:
  appnope
  backcall
  decorator
  enum34
  ipython
  ipython-genutils
  jedi>=0.10
  parso>=0.2.0
  pathlib2
  pexpect
  pickleshare
  prompt-toolkit<2.0.0,>=1.0.15
  ptyprocess>=0.5
  pygments
  simplegeneric>0.8
  six>=1.9.0
  traitlets>=4.2
  typing
  wcwidth

Finding the best candidates:
  found candidate appnope==0.1.0 (constraint was <any>)
  found candidate backcall==0.1.0 (constraint was <any>)
  found candidate decorator==4.3.0 (constraint was <any>)
  found candidate enum34==1.1.6 (constraint was <any>)
  found candidate ipython==6.3.1 (constraint was <any>)
  found candidate ipython-genutils==0.2.0 (constraint was <any>)
  found candidate jedi==0.12.0 (constraint was >=0.10)
  found candidate parso==0.2.0 (constraint was >=0.2.0)
  found candidate pathlib2==2.3.0 (constraint was <any>)
  found candidate pexpect==4.5.0 (constraint was <any>)
  found candidate pickleshare==0.7.4 (constraint was <any>)
  found candidate prompt-toolkit==1.0.15 (constraint was >=1.0.15,<2.0.0)
  found candidate ptyprocess==0.5.2 (constraint was >=0.5)
  found candidate pygments==2.2.0 (constraint was <any>)
  found candidate simplegeneric==0.8.1 (constraint was >0.8)
  found candidate six==1.11.0 (constraint was >=1.9.0)
  found candidate traitlets==4.3.2 (constraint was >=4.2)
  found candidate typing==3.6.4 (constraint was <any>)
  found candidate wcwidth==0.1.7 (constraint was <any>)

Finding secondary dependencies:
  wcwidth==0.1.7            requires -
  pygments==2.2.0           requires -
  pickleshare==0.7.4        requires pathlib2; python_version in "2.6 2.7 3.2 3.3"
  backcall==0.1.0           requires -
  ptyprocess==0.5.2         requires -
  decorator==4.3.0          requires -
  parso==0.2.0              requires -
  six==1.11.0               requires -
  prompt-toolkit==1.0.15    requires six>=1.9.0, wcwidth
  jedi==0.12.0              requires parso>=0.2.0
  traitlets==4.3.2          requires decorator, enum34; python_version == "2.7", ipython-genutils, six
  ipython-genutils==0.2.0   requires -
  simplegeneric==0.8.1      requires -
  ipython==6.3.1            requires appnope; sys_platform == "darwin", backcall, decorator, jedi>=0.10, pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2, typing; python_version <= "3.4"

New dependencies found in this round:
  adding [u'scandir', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 3: not stable

                          ROUND 4
Current constraints:
  appnope
  backcall
  decorator
  enum34
  ipython
  ipython-genutils
  jedi>=0.10
  parso>=0.2.0
  pathlib2
  pexpect
  pickleshare
  prompt-toolkit<2.0.0,>=1.0.15
  ptyprocess>=0.5
  pygments
  scandir; python_version < "3.5"
  simplegeneric>0.8
  six>=1.9.0
  traitlets>=4.2
  typing
  wcwidth

Finding the best candidates:
  found candidate appnope==0.1.0 (constraint was <any>)
  found candidate backcall==0.1.0 (constraint was <any>)
  found candidate decorator==4.3.0 (constraint was <any>)
  found candidate enum34==1.1.6 (constraint was <any>)
  found candidate ipython==6.3.1 (constraint was <any>)
  found candidate ipython-genutils==0.2.0 (constraint was <any>)
  found candidate jedi==0.12.0 (constraint was >=0.10)
  found candidate parso==0.2.0 (constraint was >=0.2.0)
  found candidate pathlib2==2.3.0 (constraint was <any>)
  found candidate pexpect==4.5.0 (constraint was <any>)
  found candidate pickleshare==0.7.4 (constraint was <any>)
  found candidate prompt-toolkit==1.0.15 (constraint was >=1.0.15,<2.0.0)
  found candidate ptyprocess==0.5.2 (constraint was >=0.5)
  found candidate pygments==2.2.0 (constraint was <any>)
  found candidate scandir==1.7 (constraint was <any>)
  found candidate simplegeneric==0.8.1 (constraint was >0.8)
  found candidate six==1.11.0 (constraint was >=1.9.0)
  found candidate traitlets==4.3.2 (constraint was >=4.2)
  found candidate typing==3.6.4 (constraint was <any>)
  found candidate wcwidth==0.1.7 (constraint was <any>)

Finding secondary dependencies:
  pickleshare==0.7.4        requires pathlib2; python_version in "2.6 2.7 3.2 3.3"
  decorator==4.3.0          requires -
  ipython==6.3.1            requires appnope; sys_platform == "darwin", backcall, decorator, jedi>=0.10, pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2, typing; python_version <= "3.4"
  traitlets==4.3.2          requires decorator, enum34; python_version == "2.7", ipython-genutils, six
  ptyprocess==0.5.2         requires -
  six==1.11.0               requires -
  jedi==0.12.0              requires parso>=0.2.0
  ipython-genutils==0.2.0   requires -
  prompt-toolkit==1.0.15    requires six>=1.9.0, wcwidth
  pygments==2.2.0           requires -
  parso==0.2.0              requires -
  simplegeneric==0.8.1      requires -
  wcwidth==0.1.7            requires -
  backcall==0.1.0           requires -
------------------------------------------------------------
Result of round 4: stable, done

Updated Pipfile.lock (ddddcd)!

Pipenv lock sets the ipython version to 6.3.1 when it should be 5.6.0. Worth noting this actually works in pipenv 10.x but not 11.

@techalchemy

This comment has been minimized.

Member

techalchemy commented Apr 19, 2018

Ah, so the bug is that you are running python 2.7, and the last version of ipython to support python 2.7 was version 5.6.0, which is why version 6.3.1 is incorrect.

What if you run pipenv lock --clear? If not, can you provide pipenv-resolver --debug ipython (it will be a lot of info)

@rainyday

This comment has been minimized.

rainyday commented Apr 19, 2018

$ pipenv lock --clear
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (ddddcd)!
$ grep -A7 \"ipython\" < Pipfile.lock
        "ipython": {
            "hashes": [
                "sha256:85882f97d75122ff8cdfe129215a408085a26039527110c8d4a2b8a5e45b7639",
                "sha256:a6ac981381b3f5f604b37a293369963485200e3639fb0404fa76092383c10c41"
            ],
            "index": "pypi",
            "version": "==6.3.1"
        },

pipenv-resolver output: https://gist.github.com/rainyday/a56c67a0635318ce6697aea5e76c2112

This happens with any package that doesn't support python 2.7 after a certain version such as pytest-quotes, django and so on.

@rainyday

This comment has been minimized.

rainyday commented Apr 20, 2018

@techalchemy would you consider reopening it as it is still an issue?

@techalchemy

This comment has been minimized.

Member

techalchemy commented Apr 25, 2018

Looks like it’s grabbing the sdists and all. For the record the virtualenv python version should be all that matters. Are you using Homebrew at all? If so you should remove any pipenv installs you did with homebrew and try pip install —user —upgrade —pre pipenv. Ideally use pip 10 then also.

@techalchemy techalchemy reopened this Apr 25, 2018

@rainyday

This comment has been minimized.

rainyday commented Apr 25, 2018

Using pyenv to install Pipenv. Tried the prerelease and pip 10 with the same results:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
ipython = "*"

[dev-packages]

[requires]
python_version = "2.7"
$ pip --version && pipenv --version && pipenv lock --clear && grep -A7 \"ipython\" < Pipfile.lock
pip 10.0.1 from /Users/<user>/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pip (python 3.6)
pipenv, version 11.10.1.dev5
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (ddddcd)!
        "ipython": {
            "hashes": [
                "sha256:85882f97d75122ff8cdfe129215a408085a26039527110c8d4a2b8a5e45b7639",
                "sha256:a6ac981381b3f5f604b37a293369963485200e3639fb0404fa76092383c10c41"
            ],
            "index": "pypi",
            "version": "==6.3.1"
        },
@techalchemy

This comment has been minimized.

Member

techalchemy commented Apr 25, 2018

I actually saw this on windows today too. I have no idea why. I don’t think we are respecting python_requires for some reason

@frostming

This comment has been minimized.

Collaborator

frostming commented Apr 26, 2018

@techalchemy @rainyday

Here are some investigations I did and I hope they help:

The magic happens when switching the ignore_compatibility flag to False in https://github.com/pypa/pipenv/blob/master/pipenv/patched/notpip/index.py#L645. However, it is never overwritten from the grep result.

It is an easy fix but I am afraid it will break things since it will generate lock files very different from existing ones.

If it is intentional that lock files are only guaranteed to work on specific environment, then we should remove all package versions that are incompatible with current environment from lock file.

The lock file content after fix:

        "ipython": {
            "hashes": [
                "sha256:8ba332d1ac80261e176e737e934fade9b252fc3adb32afc7252ccc9cb249e601",
                "sha256:c3efde0784b422538959a048d63a283f6a73d9cf5ec51b80600a8d382db0d59e",
                "sha256:d778329d0a12d084e08661ac147df99df8a64ab4400632a886449a9c444f2b6e"
            ],
            "index": "pypi",
            "version": "==5.6.0"
        },

I am preparing a PR but I am stuck in adding testing cases: mocked PyPI doesn't have requires_python metadata.

@techalchemy

This comment has been minimized.

Member

techalchemy commented Apr 26, 2018

Hey sorry for being slow. What we settled on was doing the best we can to be cross platform but single python version (so we still need this)

@techalchemy

This comment has been minimized.

Member

techalchemy commented Apr 26, 2018

Also @frostming I believe this is actually caused by there being an sdist since there’s only a python 3 wheel. We just need to crack open the sdist and check the python requires. We do this... somewhere...

@techalchemy techalchemy added this to the 11.11.0 milestone Apr 29, 2018

techalchemy added a commit that referenced this issue May 26, 2018

Patch piptools to use current environment python
- Fixes #2088, #2234, #1901
- Fully leverage piptools' compile functionality by using constraints
  in the same `RequirementSet` during resolution
- Use `PIP_PYTHON_PATH` for compatibility check to filter out
  `requires_python` markers
- Fix vcs resolution
- Update JSON API endpoints
- Enhance resolution for editable dependencies
- Minor fix for adding packages to pipfiles

Signed-off-by: Dan Ryan <dan@danryan.co>

techalchemy added a commit that referenced this issue May 26, 2018

Patch piptools to use current environment python
- Fixes #2088, #2234, #1901
- Fully leverage piptools' compile functionality by using constraints
  in the same `RequirementSet` during resolution
- Use `PIP_PYTHON_PATH` for compatibility check to filter out
  `requires_python` markers
- Fix vcs resolution
- Update JSON API endpoints
- Enhance resolution for editable dependencies
- Minor fix for adding packages to pipfiles

Signed-off-by: Dan Ryan <dan@danryan.co>

techalchemy added a commit that referenced this issue May 27, 2018

Patch piptools to use current environment python
- Fixes #2088, #2234, #1901
- Fully leverage piptools' compile functionality by using constraints
  in the same `RequirementSet` during resolution
- Use `PIP_PYTHON_PATH` for compatibility check to filter out
  `requires_python` markers
- Fix vcs resolution
- Update JSON API endpoints
- Enhance resolution for editable dependencies
- Minor fix for adding packages to pipfiles

Signed-off-by: Dan Ryan <dan@danryan.co>

@uranusjr uranusjr closed this Jun 6, 2018

@uranusjr

This comment has been minimized.

Member

uranusjr commented Jun 6, 2018

Fixed, I believe.

@rainyday

This comment has been minimized.

rainyday commented Jun 25, 2018

@uranusjr @techalchemy were these fixes deployed to 2018.6.25? Because if so it looks like they still haven't solved this :(

Using the same ipython Pipfile example from above:

$ pipenv lock --clear -v
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
using sources: [{u'url': u'https://pypi.python.org/simple', u'verify_ssl': True, u'name': u'pypi'}]
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:
  ipython (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-dZ2Ugs-requirements/pipenv-AEx6XM-constraints.txt (line 2))

Finding the best candidates:
  found candidate ipython==6.4.0 (constraint was <any>)

Finding secondary dependencies:
  ipython==6.4.0 not in cache, need to check index
  ipython==6.4.0            requires appnope; sys_platform == "darwin", backcall, decorator, enum34; python_version == "2.7", ipython-genutils, ipython==6.4.0, jedi>=0.10, parso>=0.2.0, pathlib2; python_version in "2.6 2.7 3.2 3.3", pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, ptyprocess>=0.5, pygments, scandir; python_version < "3.5", setuptools>=18.5, simplegeneric>0.8, six>=1.9.0, traitlets>=4.2, typing; python_version <= "3.4", wcwidth

New dependencies found in this round:
  adding [u'appnope', '', '[]']
  adding [u'backcall', '', '[]']
  adding [u'decorator', '', '[]']
  adding [u'enum34', '', '[]']
  adding [u'ipython', '==6.4.0', '[]']
  adding [u'ipython-genutils', '', '[]']
  adding [u'jedi', '>=0.10', '[]']
  adding [u'parso', '>=0.2.0', '[]']
  adding [u'pathlib2', '', '[]']
  adding [u'pexpect', '', '[]']
  adding [u'pickleshare', '', '[]']
  adding [u'prompt-toolkit', '<2.0.0,>=1.0.15', '[]']
  adding [u'ptyprocess', '>=0.5', '[]']
  adding [u'pygments', '', '[]']
  adding [u'scandir', '', '[]']
  adding [u'simplegeneric', '>0.8', '[]']
  adding [u'six', '>=1.9.0', '[]']
  adding [u'traitlets', '>=4.2', '[]']
  adding [u'typing', '', '[]']
  adding [u'wcwidth', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  appnope
  backcall
  decorator
  enum34
  ipython==6.4.0 (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-dZ2Ugs-requirements/pipenv-AEx6XM-constraints.txt (line 2))
  ipython-genutils
  jedi>=0.10
  parso>=0.2.0
  pathlib2
  pexpect
  pickleshare
  prompt-toolkit<2.0.0,>=1.0.15
  ptyprocess>=0.5
  pygments
  scandir
  simplegeneric>0.8
  six>=1.9.0
  traitlets>=4.2
  typing
  wcwidth

Finding the best candidates:
  found candidate appnope==0.1.0 (constraint was <any>)
  found candidate backcall==0.1.0 (constraint was <any>)
  found candidate decorator==4.3.0 (constraint was <any>)
  found candidate enum34==1.1.6 (constraint was <any>)
  found candidate ipython==6.4.0 (constraint was ==6.4.0)
  found candidate ipython-genutils==0.2.0 (constraint was <any>)
  found candidate jedi==0.12.0 (constraint was >=0.10)
  found candidate parso==0.2.1 (constraint was >=0.2.0)
  found candidate pathlib2==2.3.2 (constraint was <any>)
  found candidate pexpect==4.6.0 (constraint was <any>)
  found candidate pickleshare==0.7.4 (constraint was <any>)
  found candidate prompt-toolkit==1.0.15 (constraint was >=1.0.15,<2.0.0)
  found candidate ptyprocess==0.6.0 (constraint was >=0.5)
  found candidate pygments==2.2.0 (constraint was <any>)
  found candidate scandir==1.7 (constraint was <any>)
  found candidate simplegeneric==0.8.1 (constraint was >0.8)
  found candidate six==1.11.0 (constraint was >=1.9.0)
  found candidate traitlets==4.3.2 (constraint was >=4.2)
  found candidate typing==3.6.4 (constraint was <any>)
  found candidate wcwidth==0.1.7 (constraint was <any>)

Finding secondary dependencies:
  simplegeneric==0.8.1 not in cache, need to check index
  simplegeneric==0.8.1      requires simplegeneric==0.8.1
  pygments==2.2.0 not in cache, need to check index
  pygments==2.2.0           requires pygments==2.2.0
  backcall==0.1.0 not in cache, need to check index
  backcall==0.1.0           requires backcall==0.1.0
  ipython==6.4.0            requires appnope; sys_platform == "darwin", backcall, decorator, enum34; python_version == "2.7", ipython-genutils, ipython==6.4.0, jedi>=0.10, parso>=0.2.0, pathlib2; python_version in "2.6 2.7 3.2 3.3", pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, ptyprocess>=0.5, pygments, scandir; python_version < "3.5", setuptools>=18.5, simplegeneric>0.8, six>=1.9.0, traitlets>=4.2, typing; python_version <= "3.4", wcwidth
  ptyprocess==0.6.0 not in cache, need to check index
  ptyprocess==0.6.0         requires ptyprocess==0.6.0
  parso==0.2.1 not in cache, need to check index
  parso==0.2.1              requires parso==0.2.1
  traitlets==4.3.2 not in cache, need to check index
  traitlets==4.3.2          requires decorator, enum34; python_version == "2.7", ipython-genutils, six, traitlets==4.3.2
  jedi==0.12.0 not in cache, need to check index
  jedi==0.12.0              requires jedi==0.12.0, parso>=0.2.0
  pickleshare==0.7.4 not in cache, need to check index
  pickleshare==0.7.4        requires pathlib2; python_version in "2.6 2.7 3.2 3.3", pickleshare==0.7.4, scandir; python_version < "3.5", six
  prompt-toolkit==1.0.15 not in cache, need to check index
  prompt-toolkit==1.0.15    requires prompt-toolkit==1.0.15, six>=1.9.0, wcwidth
  wcwidth==0.1.7 not in cache, need to check index
  wcwidth==0.1.7            requires wcwidth==0.1.7
  decorator==4.3.0 not in cache, need to check index
  decorator==4.3.0          requires decorator==4.3.0
  ipython-genutils==0.2.0 not in cache, need to check index
  ipython-genutils==0.2.0   requires ipython-genutils==0.2.0
  six==1.11.0 not in cache, need to check index
  six==1.11.0               requires six==1.11.0

New dependencies found in this round:
  adding [u'appnope', '==0.1.0', '[]']
  adding [u'backcall', '==0.1.0', '[]']
  adding [u'decorator', '==4.3.0', '[]']
  adding [u'enum34', '==1.1.6', '[]']
  adding [u'ipython-genutils', '==0.2.0', '[]']
  adding [u'jedi', '==0.12.0,>=0.10', '[]']
  adding [u'parso', '==0.2.1,>=0.2.0', '[]']
  adding [u'pathlib2', '==2.3.2', '[]']
  adding [u'pexpect', '==4.6.0', '[]']
  adding [u'pickleshare', '==0.7.4', '[]']
  adding [u'prompt-toolkit', '<2.0.0,==1.0.15,>=1.0.15', '[]']
  adding [u'ptyprocess', '==0.6.0,>=0.5', '[]']
  adding [u'pygments', '==2.2.0', '[]']
  adding [u'scandir', '==1.7', '[]']
  adding [u'simplegeneric', '==0.8.1,>0.8', '[]']
  adding [u'six', '==1.11.0,>=1.9.0', '[]']
  adding [u'traitlets', '==4.3.2,>=4.2', '[]']
  adding [u'typing', '==3.6.4', '[]']
  adding [u'wcwidth', '==0.1.7', '[]']
Removed dependencies in this round:
  removing [u'appnope', '', '[]']
  removing [u'backcall', '', '[]']
  removing [u'decorator', '', '[]']
  removing [u'enum34', '', '[]']
  removing [u'ipython-genutils', '', '[]']
  removing [u'jedi', '>=0.10', '[]']
  removing [u'parso', '>=0.2.0', '[]']
  removing [u'pathlib2', '', '[]']
  removing [u'pexpect', '', '[]']
  removing [u'pickleshare', '', '[]']
  removing [u'prompt-toolkit', '<2.0.0,>=1.0.15', '[]']
  removing [u'ptyprocess', '>=0.5', '[]']
  removing [u'pygments', '', '[]']
  removing [u'scandir', '', '[]']
  removing [u'simplegeneric', '>0.8', '[]']
  removing [u'six', '>=1.9.0', '[]']
  removing [u'traitlets', '>=4.2', '[]']
  removing [u'typing', '', '[]']
  removing [u'wcwidth', '', '[]']
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 2: not stable

                          ROUND 3
Current constraints:
  appnope==0.1.0
  backcall==0.1.0
  decorator==4.3.0
  enum34==1.1.6
  ipython==6.4.0 (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-dZ2Ugs-requirements/pipenv-AEx6XM-constraints.txt (line 2))
  ipython-genutils==0.2.0
  jedi==0.12.0,>=0.10
  parso==0.2.1,>=0.2.0
  pathlib2==2.3.2
  pexpect==4.6.0
  pickleshare==0.7.4
  prompt-toolkit<2.0.0,==1.0.15,>=1.0.15
  ptyprocess==0.6.0,>=0.5
  pygments==2.2.0
  scandir==1.7
  simplegeneric==0.8.1,>0.8
  six==1.11.0,>=1.9.0
  traitlets==4.3.2,>=4.2
  typing==3.6.4
  wcwidth==0.1.7

Finding the best candidates:
  found candidate appnope==0.1.0 (constraint was ==0.1.0)
  found candidate backcall==0.1.0 (constraint was ==0.1.0)
  found candidate decorator==4.3.0 (constraint was ==4.3.0)
  found candidate enum34==1.1.6 (constraint was ==1.1.6)
  found candidate ipython==6.4.0 (constraint was ==6.4.0)
  found candidate ipython-genutils==0.2.0 (constraint was ==0.2.0)
  found candidate jedi==0.12.0 (constraint was >=0.10,==0.12.0)
  found candidate parso==0.2.1 (constraint was >=0.2.0,==0.2.1)
  found candidate pathlib2==2.3.2 (constraint was ==2.3.2)
  found candidate pexpect==4.6.0 (constraint was ==4.6.0)
  found candidate pickleshare==0.7.4 (constraint was ==0.7.4)
  found candidate prompt-toolkit==1.0.15 (constraint was ==1.0.15,>=1.0.15,<2.0.0)
  found candidate ptyprocess==0.6.0 (constraint was >=0.5,==0.6.0)
  found candidate pygments==2.2.0 (constraint was ==2.2.0)
  found candidate scandir==1.7 (constraint was ==1.7)
  found candidate simplegeneric==0.8.1 (constraint was >0.8,==0.8.1)
  found candidate six==1.11.0 (constraint was ==1.11.0,>=1.9.0)
  found candidate traitlets==4.3.2 (constraint was >=4.2,==4.3.2)
  found candidate typing==3.6.4 (constraint was ==3.6.4)
  found candidate wcwidth==0.1.7 (constraint was ==0.1.7)

Finding secondary dependencies:
  pickleshare==0.7.4        requires pathlib2; python_version in "2.6 2.7 3.2 3.3", pickleshare==0.7.4, scandir; python_version < "3.5", six
  backcall==0.1.0           requires backcall==0.1.0
  pygments==2.2.0           requires pygments==2.2.0
  ipython-genutils==0.2.0   requires ipython-genutils==0.2.0
  six==1.11.0               requires six==1.11.0
  prompt-toolkit==1.0.15    requires prompt-toolkit==1.0.15, six>=1.9.0, wcwidth
  parso==0.2.1              requires parso==0.2.1
  wcwidth==0.1.7            requires wcwidth==0.1.7
  decorator==4.3.0          requires decorator==4.3.0
  ptyprocess==0.6.0         requires ptyprocess==0.6.0
  jedi==0.12.0              requires jedi==0.12.0, parso>=0.2.0
  simplegeneric==0.8.1      requires simplegeneric==0.8.1
  ipython==6.4.0            requires appnope; sys_platform == "darwin", backcall, decorator, enum34; python_version == "2.7", ipython-genutils, ipython==6.4.0, jedi>=0.10, parso>=0.2.0, pathlib2; python_version in "2.6 2.7 3.2 3.3", pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, ptyprocess>=0.5, pygments, scandir; python_version < "3.5", setuptools>=18.5, simplegeneric>0.8, six>=1.9.0, traitlets>=4.2, typing; python_version <= "3.4", wcwidth
  traitlets==4.3.2          requires decorator, enum34; python_version == "2.7", ipython-genutils, six, traitlets==4.3.2
------------------------------------------------------------
Result of round 3: stable, done

Updated Pipfile.lock (ddddcd)!

@uranusjr uranusjr removed this from the 11.11.0 milestone Jun 26, 2018

@frostming

This comment has been minimized.

Collaborator

frostming commented Jun 26, 2018

Seems the fix was reverted in c31a311. @techalchemy , was it by mistake?

A test case also needs to be added

@techalchemy

This comment has been minimized.

Member

techalchemy commented Jun 26, 2018

Test case for this specifically was a bit difficult, if you can come up with one I’d appreciate it. I think I see the line that’s missing which needs to be included. I’ll check when I get to the office

@frostming

This comment has been minimized.

Collaborator

frostming commented Jun 26, 2018

@techalchemy OK, I just come up with an idea that we can mock sys.version_info or something to return 2.7 forcely.

@techalchemy

This comment has been minimized.

Member

techalchemy commented Jun 26, 2018

ugh, we used to do that but it was not fun.

I meant to fix this, do I get points for that ?! :p

@techalchemy techalchemy reopened this Jun 26, 2018

techalchemy added a commit that referenced this issue Jun 26, 2018

Updated piptools to only lock compatible packages
- Fixes #1901

Signed-off-by: Dan Ryan <dan@danryan.co>

techalchemy added a commit that referenced this issue Jun 27, 2018

Updated piptools to only lock compatible packages
- Fixes #1901

Signed-off-by: Dan Ryan <dan@danryan.co>

techalchemy added a commit that referenced this issue Jun 27, 2018

@rainyday

This comment has been minimized.

rainyday commented Jul 2, 2018

tested and it is in today's release. thanks for your hard work guys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment