Pipfile python_version not taken into account when generating lockfile

Pipenv lock/install/sync generates a Pipfile.lock with package versions incompatible with the version of Python

output of python -m pipenv.help: https://pastebin.com/raw/r4WCG5AR

---

Expected result

Pipenv generates a lockfile with installable packages and then installs them.

Actual result

Pipenv generates a Pipfile.lock containing requirements that are incompatible with the version of Python specified in Pipfile using python_version or python_full_version and installation fails. Example:

Error message:

Collecting pylint-quotes==0.1.9 (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-_auxxp4x-requirements/pipenv-u4w5lmul-requirement.txt (line 1))
  1 location(s) to search for versions of pylint-quotes:
  * https://pypi.python.org/simple/pylint-quotes/
  Getting page https://pypi.python.org/simple/pylint-quotes/
  Looking up "https://pypi.python.org/simple/pylint-quotes/" in the cache
  Current age based on date: 11
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  The response is "fresh", returning cached response
  600 > 11
  Analyzing links from page https://pypi.python.org/simple/pylint-quotes/
    Found link https://pypi.python.org/packages/7c/88/7a894a269a58a907fe173282f50d642dcbaa44a0b5d12043f4785347f11b/pylint-quotes-0.1.3.tar.gz#md5=2351ca090248406e0eeafa75d7a1eb10 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.3
    Found link https://pypi.python.org/packages/a0/0a/372a8219a6a42a9fdba7bdc00aaffb3a9d88d01138022d76c08f6b83764a/pylint_quotes-0.1.3-py2.py3-none-any.whl#md5=1097eeffc8f0a75cc843cc0c97866be7 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.3
    Found link https://pypi.python.org/packages/7d/a7/b1ddde1d38c1141ab2d7c51d307ef7f81d62b970fc035ea8db74911821c4/pylint-quotes-0.1.5.tar.gz#md5=1cd3ebbcfc518653c81593d2486da0d0 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.5
    Found link https://pypi.python.org/packages/18/69/f19c51efaf3951fb570a78243b261a7bf178e66468f3692e1cd225fb4cfb/pylint_quotes-0.1.5-py2.py3-none-any.whl#md5=5cc78a567e4c25aa300c4bf933819e8b (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.5
    Found link https://pypi.python.org/packages/8c/c4/04c56b0446962f6c7424d76d5799747d40ad3472684e8995ba01a3fcdbdc/pylint-quotes-0.1.6.tar.gz#md5=86f79ae93e06a6fc488c7f73924be812 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.6
    Found link https://pypi.python.org/packages/e7/ad/b3037ce9414ab986052d395b1b9cb6026e6e296afade18b2840d1363a073/pylint_quotes-0.1.6-py2.py3-none-any.whl#md5=1213fed2daec3507c9207d356de72b06 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.6
    Found link https://pypi.python.org/packages/8d/ef/a135a29c4c4358b60712ef44b2eac21dda876ab9d0203bb4f254800d4255/pylint-quotes-0.1.7.tar.gz#md5=389d3ec970ef4b8d42d8247df3d08373 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.7
    Found link https://pypi.python.org/packages/e8/b5/f03009ab72fe1d1442c4b8c3fc06680ecb4c1f0385b5b0bf2bf221b3195e/pylint_quotes-0.1.7-py2.py3-none-any.whl#md5=dea6d20e390fe80013d8a33be932d206 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.7
    Found link https://pypi.python.org/packages/1c/b9/e14468554607a685abcb0d617d1ef7c5f25e5cbb1e280fed370594993006/pylint-quotes-0.1.8.tar.gz#md5=7330dcdb88e941ad2f03048ba95ede05 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.8
    Found link https://pypi.python.org/packages/28/f5/e00a9b7597ba7b92b58ddac472f9cf5bfd347022a65a4beafd5f682a7758/pylint_quotes-0.1.8-py2.py3-none-any.whl#md5=fb713d376b27abdbd9491b126e85b5a5 (from https://pypi.python.org/simple/pylint-quotes/), version: 0.1.8
    The package https://pypi.python.org/packages/ba/20/4e59a01fd90fbe67385b790cb6ddb4a462f4d8e15548bab27c59c3b23fd3/pylint-quotes-0.1.9.tar.gz#md5=02eaa4853463f8f867f36e031b2503f0 (from https://pypi.python.org/simple/pylint-quotes/) (requires-python:>=3.4) is incompatible with the pythonversion in use. Acceptable python versions are:>=3.4
    The package https://pypi.python.org/packages/c2/cc/afc270fa43a12d10736422ddaa653c7c7664561f10b182b76884438637c2/pylint_quotes-0.1.9-py2.py3-none-any.whl#md5=7c680717a5af1c59e68809c865478f82 (from https://pypi.python.org/simple/pylint-quotes/) (requires-python:>=3.4) is incompatible with the pythonversion in use. Acceptable python versions are:>=3.4
  Could not find a version that satisfies the requirement pylint-quotes==0.1.9 (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-_auxxp4x-requirements/pipenv-u4w5lmul-requirement.txt (line 1)) (from versions: 0.1.3, 0.1.5, 0.1.6, 0.1.7, 0.1.8)
Cleaning up...
No matching distribution found for pylint-quotes==0.1.9 (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-_auxxp4x-requirements/pipenv-u4w5lmul-requirement.txt (line 1))
Exception information:
Traceback (most recent call last):
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/Users/<User>/.local/share/virtualenvs/py-VEuvpTJA/lib/python2.7/site-packages/pip/index.py", line 514, in find_requirement
    'No matching distribution found for %s' % req
DistributionNotFound: No matching distribution found for pylint-quotes==0.1.9 (from -r /var/folders/rj/210lgzrd4q5ccsyby0rdxg0924842d/T/pipenv-_auxxp4x-requirements/pipenv-u4w5lmul-requirement.txt (line 1))

An error occurred while installing pylint-quotes==0.1.9! Will try again.

Steps to replicate

Create a Pipfile that specifies a python version and packages with a version range that includes versions that are both compatible and incompatible with the python_version specified.

Example Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[requires]
python_version = "2.7"
python_full_version = "2.7.14"

[packages]
beaker = "==1.9.0"
bottle = "==0.12.13"
funcsigs = "==1.0.2"
gevent = "==1.2.2"
greenlet = "==0.4.12"
influxdb = "<=6.0.0"
jsoncomment = "==0.3.0"
netaddr = "==0.7.19"
numpy = "==1.11.2"
pandas = "==0.19.1"
pyrad = "==2.1"
python-dateutil = "==2.6.1"
pytz = "==2017.2"
requests = "*"
schedule = "==0.4.2"
tacacs-plus = "==2.0"
ujson = "==1.35"
bson = "*"
pymongo = "*"
"h5py" = "*"
environs = "*"
pyyaml = "*"
deprecated = "*"
appdirs = "*"
pycryptodome = "==3.4.11"
click = "*"
pyopenssl = "*"
pem = "*"

[dev-packages]
pytest = "*"
rope = "*"
ipython = "*"
mock = "*"
pylint = "*"
pylint-quotes = "*"
yapf = "*"
teamcity-messages = "*"
webtest = "*"
requests-mock = "*"
pdbpp = "*"
pytest-mock = "*"
pytest-cov = "*"
pytest-pythonpath = "*"

With this Pipfile both pylint-quotes and ipython will fail since pylint-quotes 1.9 and ipython 6.3.0 are both incompatible with python 2.7.14

[techalchemy]

Thanks for the report, sorry for the issue :( this is a know issue and is fixed in master, closing for now. Will have a release out soon

[frostming]

This issue does exist on pipenv 11.10.0. The root cause is the resolver doesn't compare the current python version with python_requires and always picks the latest one.

Pipfile content:

[requires]
python_version = "2.7"

[packages]
django = "*"    # python2.7 requires django < 2

$ python -V
Python 2.7.14

pipenv lock verbose output

Locking [packages] dependencies…

Using pip: -i https://pypi.python.org/simple

                          ROUND 1                           
Current constraints:
  django

Finding the best candidates:
  found candidate django==2.0.4 (constraint was <any>)

Finding secondary dependencies:
  django==2.0.4 not in cache, need to check index
  django==2.0.4             requires pytz

New dependencies found in this round:
  adding [u'pytz', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  django
  pytz

Finding the best candidates:
  found candidate django==2.0.4 (constraint was <any>)
  found candidate pytz==2018.4 (constraint was <any>)

Finding secondary dependencies:
  django==2.0.4             requires pytz
  pytz==2018.4 not in cache, need to check index
  pytz==2018.4              requires -
------------------------------------------------------------
Result of round 2: stable, done


Updated Pipfile.lock (0454a4)!

This is not fixed

[techalchemy]

To clarify, the words in your pipfile are used to generate your environment; the environment itself performs resolution. I need to see the output of pipenv lock —verbose to really understand what’s going on but I’m guessing based on the small part that you included that it one of your dependencies actually requires this version

Here is a simplified Pipfile that reproduces the issue

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
ipython = "*"

[dev-packages]

[requires]
python_version = "2.7"

$ pipenv lock -vvv
Creating a virtualenv for this project…
Using /Users/<user>/.pyenv/versions/2.7.14/bin/python2.7 (2.7.14) to create virtualenv…
⠋Running virtualenv with interpreter /Users/<user>/.pyenv/versions/2.7.14/bin/python2.7
New python executable in /Users/<user>/.local/share/virtualenvs/pipenv-test-3vVgCXNF/bin/python2.7
Also creating executable in /Users/<user>/.local/share/virtualenvs/pipenv-test-3vVgCXNF/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/<user>/.local/share/virtualenvs/pipenv-test-3vVgCXNF
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:
  ipython

Finding the best candidates:
  found candidate ipython==6.3.1 (constraint was <any>)

Finding secondary dependencies:
  ipython==6.3.1            requires appnope; sys_platform == "darwin", backcall, decorator, jedi>=0.10, pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2, typing; python_version <= "3.4"

New dependencies found in this round:
  adding [u'appnope', '', '[]']
  adding [u'backcall', '', '[]']
  adding [u'decorator', '', '[]']
  adding [u'jedi', '>=0.10', '[]']
  adding [u'pexpect', '', '[]']
  adding [u'pickleshare', '', '[]']
  adding [u'prompt-toolkit', '<2.0.0,>=1.0.15', '[]']
  adding [u'pygments', '', '[]']
  adding [u'simplegeneric', '>0.8', '[]']
  adding [u'traitlets', '>=4.2', '[]']
  adding [u'typing', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  appnope
  backcall
  decorator
  ipython
  jedi>=0.10
  pexpect
  pickleshare
  prompt-toolkit<2.0.0,>=1.0.15
  pygments
  simplegeneric>0.8
  traitlets>=4.2
  typing

Finding the best candidates:
  found candidate appnope==0.1.0 (constraint was <any>)
  found candidate backcall==0.1.0 (constraint was <any>)
  found candidate decorator==4.3.0 (constraint was <any>)
  found candidate ipython==6.3.1 (constraint was <any>)
  found candidate jedi==0.12.0 (constraint was >=0.10)
  found candidate pexpect==4.5.0 (constraint was <any>)
  found candidate pickleshare==0.7.4 (constraint was <any>)
  found candidate prompt-toolkit==1.0.15 (constraint was >=1.0.15,<2.0.0)
  found candidate pygments==2.2.0 (constraint was <any>)
  found candidate simplegeneric==0.8.1 (constraint was >0.8)
  found candidate traitlets==4.3.2 (constraint was >=4.2)
  found candidate typing==3.6.4 (constraint was <any>)

Finding secondary dependencies:
  pygments==2.2.0           requires -
  prompt-toolkit==1.0.15    requires six>=1.9.0, wcwidth
  decorator==4.3.0          requires -
  jedi==0.12.0              requires parso>=0.2.0
  backcall==0.1.0           requires -
  traitlets==4.3.2          requires decorator, enum34; python_version == "2.7", ipython-genutils, six
  pickleshare==0.7.4        requires pathlib2; python_version in "2.6 2.7 3.2 3.3"
  ipython==6.3.1            requires appnope; sys_platform == "darwin", backcall, decorator, jedi>=0.10, pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2, typing; python_version <= "3.4"
  simplegeneric==0.8.1      requires -

New dependencies found in this round:
  adding [u'enum34', '', '[]']
  adding [u'ipython-genutils', '', '[]']
  adding [u'parso', '>=0.2.0', '[]']
  adding [u'pathlib2', '', '[]']
  adding [u'ptyprocess', '>=0.5', '[]']
  adding [u'six', '>=1.9.0', '[]']
  adding [u'wcwidth', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 2: not stable

                          ROUND 3
Current constraints:
  appnope
  backcall
  decorator
  enum34
  ipython
  ipython-genutils
  jedi>=0.10
  parso>=0.2.0
  pathlib2
  pexpect
  pickleshare
  prompt-toolkit<2.0.0,>=1.0.15
  ptyprocess>=0.5
  pygments
  simplegeneric>0.8
  six>=1.9.0
  traitlets>=4.2
  typing
  wcwidth

Finding the best candidates:
  found candidate appnope==0.1.0 (constraint was <any>)
  found candidate backcall==0.1.0 (constraint was <any>)
  found candidate decorator==4.3.0 (constraint was <any>)
  found candidate enum34==1.1.6 (constraint was <any>)
  found candidate ipython==6.3.1 (constraint was <any>)
  found candidate ipython-genutils==0.2.0 (constraint was <any>)
  found candidate jedi==0.12.0 (constraint was >=0.10)
  found candidate parso==0.2.0 (constraint was >=0.2.0)
  found candidate pathlib2==2.3.0 (constraint was <any>)
  found candidate pexpect==4.5.0 (constraint was <any>)
  found candidate pickleshare==0.7.4 (constraint was <any>)
  found candidate prompt-toolkit==1.0.15 (constraint was >=1.0.15,<2.0.0)
  found candidate ptyprocess==0.5.2 (constraint was >=0.5)
  found candidate pygments==2.2.0 (constraint was <any>)
  found candidate simplegeneric==0.8.1 (constraint was >0.8)
  found candidate six==1.11.0 (constraint was >=1.9.0)
  found candidate traitlets==4.3.2 (constraint was >=4.2)
  found candidate typing==3.6.4 (constraint was <any>)
  found candidate wcwidth==0.1.7 (constraint was <any>)

Finding secondary dependencies:
  wcwidth==0.1.7            requires -
  pygments==2.2.0           requires -
  pickleshare==0.7.4        requires pathlib2; python_version in "2.6 2.7 3.2 3.3"
  backcall==0.1.0           requires -
  ptyprocess==0.5.2         requires -
  decorator==4.3.0          requires -
  parso==0.2.0              requires -
  six==1.11.0               requires -
  prompt-toolkit==1.0.15    requires six>=1.9.0, wcwidth
  jedi==0.12.0              requires parso>=0.2.0
  traitlets==4.3.2          requires decorator, enum34; python_version == "2.7", ipython-genutils, six
  ipython-genutils==0.2.0   requires -
  simplegeneric==0.8.1      requires -
  ipython==6.3.1            requires appnope; sys_platform == "darwin", backcall, decorator, jedi>=0.10, pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2, typing; python_version <= "3.4"

New dependencies found in this round:
  adding [u'scandir', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 3: not stable

                          ROUND 4
Current constraints:
  appnope
  backcall
  decorator
  enum34
  ipython
  ipython-genutils
  jedi>=0.10
  parso>=0.2.0
  pathlib2
  pexpect
  pickleshare
  prompt-toolkit<2.0.0,>=1.0.15
  ptyprocess>=0.5
  pygments
  scandir; python_version < "3.5"
  simplegeneric>0.8
  six>=1.9.0
  traitlets>=4.2
  typing
  wcwidth

Finding the best candidates:
  found candidate appnope==0.1.0 (constraint was <any>)
  found candidate backcall==0.1.0 (constraint was <any>)
  found candidate decorator==4.3.0 (constraint was <any>)
  found candidate enum34==1.1.6 (constraint was <any>)
  found candidate ipython==6.3.1 (constraint was <any>)
  found candidate ipython-genutils==0.2.0 (constraint was <any>)
  found candidate jedi==0.12.0 (constraint was >=0.10)
  found candidate parso==0.2.0 (constraint was >=0.2.0)
  found candidate pathlib2==2.3.0 (constraint was <any>)
  found candidate pexpect==4.5.0 (constraint was <any>)
  found candidate pickleshare==0.7.4 (constraint was <any>)
  found candidate prompt-toolkit==1.0.15 (constraint was >=1.0.15,<2.0.0)
  found candidate ptyprocess==0.5.2 (constraint was >=0.5)
  found candidate pygments==2.2.0 (constraint was <any>)
  found candidate scandir==1.7 (constraint was <any>)
  found candidate simplegeneric==0.8.1 (constraint was >0.8)
  found candidate six==1.11.0 (constraint was >=1.9.0)
  found candidate traitlets==4.3.2 (constraint was >=4.2)
  found candidate typing==3.6.4 (constraint was <any>)
  found candidate wcwidth==0.1.7 (constraint was <any>)

Finding secondary dependencies:
  pickleshare==0.7.4        requires pathlib2; python_version in "2.6 2.7 3.2 3.3"
  decorator==4.3.0          requires -
  ipython==6.3.1            requires appnope; sys_platform == "darwin", backcall, decorator, jedi>=0.10, pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.15, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2, typing; python_version <= "3.4"
  traitlets==4.3.2          requires decorator, enum34; python_version == "2.7", ipython-genutils, six
  ptyprocess==0.5.2         requires -
  six==1.11.0               requires -
  jedi==0.12.0              requires parso>=0.2.0
  ipython-genutils==0.2.0   requires -
  prompt-toolkit==1.0.15    requires six>=1.9.0, wcwidth
  pygments==2.2.0           requires -
  parso==0.2.0              requires -
  simplegeneric==0.8.1      requires -
  wcwidth==0.1.7            requires -
  backcall==0.1.0           requires -
------------------------------------------------------------
Result of round 4: stable, done

Updated Pipfile.lock (ddddcd)!

Pipenv lock sets the ipython version to 6.3.1 when it should be 5.6.0. Worth noting this actually works in pipenv 10.x but not 11.

[techalchemy]

Ah, so the bug is that you are running python 2.7, and the last version of ipython to support python 2.7 was version 5.6.0, which is why version 6.3.1 is incorrect.

What if you run pipenv lock --clear? If not, can you provide pipenv-resolver --debug ipython (it will be a lot of info)

$ pipenv lock --clear
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (ddddcd)!
$ grep -A7 \"ipython\" < Pipfile.lock
        "ipython": {
            "hashes": [
                "sha256:85882f97d75122ff8cdfe129215a408085a26039527110c8d4a2b8a5e45b7639",
                "sha256:a6ac981381b3f5f604b37a293369963485200e3639fb0404fa76092383c10c41"
            ],
            "index": "pypi",
            "version": "==6.3.1"
        },

pipenv-resolver output: https://gist.github.com/rainyday/a56c67a0635318ce6697aea5e76c2112

This happens with any package that doesn't support python 2.7 after a certain version such as pytest-quotes, django and so on.

@techalchemy would you consider reopening it as it is still an issue?

[techalchemy]

Looks like it’s grabbing the sdists and all. For the record the virtualenv python version should be all that matters. Are you using Homebrew at all? If so you should remove any pipenv installs you did with homebrew and try pip install —user —upgrade —pre pipenv. Ideally use pip 10 then also.

Using pyenv to install Pipenv. Tried the prerelease and pip 10 with the same results:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
ipython = "*"

[dev-packages]

[requires]
python_version = "2.7"

$ pip --version && pipenv --version && pipenv lock --clear && grep -A7 \"ipython\" < Pipfile.lock
pip 10.0.1 from /Users/<user>/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pip (python 3.6)
pipenv, version 11.10.1.dev5
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (ddddcd)!
        "ipython": {
            "hashes": [
                "sha256:85882f97d75122ff8cdfe129215a408085a26039527110c8d4a2b8a5e45b7639",
                "sha256:a6ac981381b3f5f604b37a293369963485200e3639fb0404fa76092383c10c41"
            ],
            "index": "pypi",
            "version": "==6.3.1"
        },

[techalchemy]

I actually saw this on windows today too. I have no idea why. I don’t think we are respecting python_requires for some reason