New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

markers 'platform_python_implementation == "cpython"' don't match your environment when installing gevent #2113

Closed
jfly opened this Issue May 1, 2018 · 19 comments

Comments

Projects
None yet
7 participants
@jfly
Contributor

jfly commented May 1, 2018

When installing gevent using pipenv, greenlet is not being installed.

$ python -m pipenv.help output

Pipenv version: '11.10.1'

Pipenv location: '/usr/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python'

Other Python installations in PATH:

  • 2.7: /usr/bin/python2.7

  • 2.7: /usr/bin/python2.7

  • 2.7: /home/jeremy/.pyenv/shims/python2.7

  • 2.7: /usr/bin/python2.7

  • 3.6: /usr/bin/python3.6m

  • 3.6: /usr/bin/python3.6

  • 3.6: /home/jeremy/.pyenv/shims/python3.6

  • 3.6: /usr/bin/python3.6

  • 3.6.5: /usr/bin/python

  • 3.6.5: /home/jeremy/.pyenv/shims/python

  • 3.6.5: /usr/bin/python

  • 2.7.14: /usr/bin/python2

  • 2.7.14: /home/jeremy/.pyenv/shims/python2

  • 2.7.14: /usr/bin/python2

  • 3.6.5: /usr/bin/python3

  • 3.6.5: /home/jeremy/.pyenv/shims/python3

  • 3.6.5: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.16.5-1-ARCH',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Thu Apr 26 16:53:40 UTC 2018',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

  • LS_COLORS
  • LANG
  • LESS
  • FZF_DEFAULT_COMMAND
  • DISPLAY
  • PYENV_ROOT
  • OLDPWD
  • INVOCATION_ID
  • EDITOR
  • COLORTERM
  • PYENV_VIRTUALENV_INIT
  • MOZ_PLUGIN_PATH
  • PYENV_HOOK_PATH
  • XDG_VTNR
  • ZSH
  • SSH_AUTH_SOCK
  • FZF_CTRL_T_COMMAND
  • XDG_SESSION_ID
  • USER
  • PYENV_DIR
  • PAGER
  • LSCOLORS
  • PWD
  • HOME
  • LC_CTYPE
  • JOURNAL_STREAM
  • PYENV_VERSION
  • MAIL
  • VISUAL
  • WINDOWPATH
  • TERM
  • SHELL
  • VTE_VERSION
  • XDG_SEAT
  • SHLVL
  • PYENV_SHELL
  • WINDOWID
  • MAVEN_OPTS
  • LOGNAME
  • DBUS_SESSION_BUS_ADDRESS
  • XDG_RUNTIME_DIR
  • XAUTHORITY
  • PATH
  • PYTHONDONTWRITEBYTECODE
  • PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /usr/bin:/home/jeremy/.pyenv/libexec:/home/jeremy/.pyenv/plugins/python-build/bin:/home/jeremy/.pyenv/plugins/pyenv-virtualenv/bin:/home/jeremy/.pyenv/plugins/pyenv-virtualenv/shims:/home/jeremy/.pyenv/shims:/home/jeremy/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jeremy/bin:/home/jeremy/.gem/ruby/2.5.0/bin:/home/jeremy/.local/bin/:/home/jeremy/thirdrepos/google-cloud-sdk/bin
  • SHELL: /usr/bin/zsh
  • EDITOR: vim
  • LANG: en_US.UTF-8
  • PWD: /home/jeremy


Expected result

I expect installing gevent to also result in greenlet being installed.

Actual result

Instead of having greenlet installed, I get a warning about "Ignoring greenlet: markers 'platform_python_implementation == "cpython"' don't match your environment".

Steps to replicate
~/tmp/pipenving echo '[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
gevent = "*"

[dev-packages]

[requires]
python_version = "2.7"

[pipenv]
allow_prereleases = true' > Pipfile
➜  ~/tmp/pipenving pipenv install
Creating a virtualenv for this project…
Using /home/jeremy/.pyenv/versions/jfly-2.7.12/bin/python2.7 (2.7.12) to create virtualenv…
⠋Running virtualenv with interpreter /home/jeremy/.pyenv/versions/jfly-2.7.12/bin/python2.7
Using real prefix '/home/jeremy/.pyenv/versions/2.7.12'
New python executable in /home/jeremy/.local/share/virtualenvs/pipenving-IYbJmdaL/bin/python2.7
Also creating executable in /home/jeremy/.local/share/virtualenvs/pipenving-IYbJmdaL/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/jeremy/.local/share/virtualenvs/pipenving-IYbJmdaL
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (6d8673)!
Installing dependencies from Pipfile.lock (6d8673)…
Ignoring greenlet: markers 'platform_python_implementation == "cpython"' don't match your environment
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell
➜  ~/tmp/pipenving pip freeze | grep gevent
You are using pip version 9.0.1, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
➜  ~/tmp/pipenving pipenv run pip freeze | grep greenlet
➜  ~/tmp/pipenving pipenv run pip freeze | grep gevent
gevent==1.3b1

Note the warning above about platform_python_implementation == "cpython" not matching my environment. I looked into this a bit, and according to PEP508, platform_python_implementation corresponds to platform.python_implementation(), which is equal to "CPython" for me:

➜  ~/tmp/pipenving pipenv run python -c "import platform; print platform.python_implementation()"
CPython

I verified that if I manually change the "markers": "platform_python_implementation == 'cpython'" line in my Pipfile.lock from "cpython" to "CPython", then pipenv install works happily:

➜  ~/tmp/pipenving sed -i 's/cpython/CPython/' Pipfile.lock 
➜  ~/tmp/pipenving pipenv install                           
Installing dependencies from Pipfile.lock (6d8673)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell
➜  ~/tmp/pipenving pipenv run pip freeze | grep greenlet
greenlet==0.4.13

However, this isn't a permanent solution for me, as subsequent runs of pipenv lock reset my Pipfile.lock to have a lowercase "cpython":

➜  ~/tmp/pipenving pipenv lock
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (6d8673)!
➜  ~/tmp/pipenving grep cpython Pipfile.lock 
            "markers": "platform_python_implementation == 'cpython'",

I confirmed that gevent 1.3b1 is correctly capitalizing "CPython" in its setup.py: https://github.com/gevent/gevent/blob/1.3b1/setup.py#L190, so I concluded that pipenv must be lowercasing this marker somewhere. I stepped through pipenv with a debugger to track down where the platform_python_implementation marker is getting lowercased, and I tracked it down to this line:

self.dependency_cache[ireq] = sorted(format_requirement(ireq) for ireq in dependencies)

which invokes format_requirement, which lowercases the entire requirement, including the markers:

line = str(ireq.req).lower()

It looks like the format_requirement method was written with the intention of markers being specified as an optional second argument, so maybe the right fix is to change our invocation of format_requirement to actually split off the markers and pass them in separately?

@uranusjr

This comment has been minimized.

Member

uranusjr commented May 2, 2018

If I read PEP 508 correctly, the comparison should be case insensitive (it even lists 'cpython' as an acceptable value). The lower-casing shouldn’t be a problem; it’s the comparing part that should be fixed.

@jfly

This comment has been minimized.

Contributor

jfly commented May 2, 2018

Are you sure about that? The only mention of case sensitivity I see in PEP 508 is about python distribution names. As I understood it, this is about environment markers, which I believe should be using regular Python string comparison (emphasis mine):

The <version_cmp> operators use the PEP-440 [4] version comparison rules when those are defined (that is when both sides have a valid version specifier). If there is no defined PEP-440 behaviour and the operator exists in Python, then the operator falls back to the Python behaviour.

The example you give of 'cpython' in the PEP is for the implementation_name environment marker. The issue I raised is about the platform_python_implementation environment marker, which PEP 508 gives "CPython, Jython" as example values.

@uranusjr

This comment has been minimized.

Member

uranusjr commented May 2, 2018

Yes, you are correct, I misread. The code you mentioned is the correct place to fix this in that case. Would you mind convert it to a pull request?

@jfly

This comment has been minimized.

Contributor

jfly commented May 2, 2018

Sure, I will try to send in a PR tomorrow. Thanks!

@techalchemy

This comment has been minimized.

Member

techalchemy commented May 2, 2018

Jeez you got into some code on this. Here’s the underlying code from the platform module:

def python_implementation():

    """ Returns a string identifying the Python implementation.
        Currently, the following implementations are identified:
          'CPython' (C implementation of Python),
          'IronPython' (.NET implementation of Python),
          'Jython' (Java implementation of Python),
          'PyPy' (Python implementation of Python).
    """
    return _sys_version()[0]

jfly added a commit to jfly/pipenv that referenced this issue May 2, 2018

When formatting a requirement, only lowercase its name.
This fixes pypa#2113.

This bug was introduced in
<jazzband/pip-tools#452> as a band-aid fix to
<jazzband/pip-tools#431>. Pipenv then copied
that code in <pypa@2553ebc#diff-b56b95ccea8595a0f6f24ea753842976>, and inherited this latent bug.

Maybe the right fix is for pypa/packaging to lowercase the name? There's
a comment here
<https://github.com/pypa/packaging/blob/16.8/packaging/requirements.py#L86>
about normalizing the requirement's name, which might be what this is
referring to.
@jfly

This comment has been minimized.

Contributor

jfly commented May 2, 2018

I've sent in a PR here: #2123.

Until we get this fixed, I am able to workaround this by adding the following line to our Pipfile:

greenlet = {version = "*", platform_python_implementation="=='CPython'"}

(I got the idea for this hack from #1757 (comment))

jfly added a commit to jfly/pipenv that referenced this issue May 7, 2018

When formatting a requirement, only lowercase its name.
This fixes pypa#2113.

This bug was introduced in
<jazzband/pip-tools#452> as a band-aid fix to
<jazzband/pip-tools#431>. Pipenv then copied
that code in <pypa@2553ebc#diff-b56b95ccea8595a0f6f24ea753842976>, and inherited this latent bug.

Maybe the right fix is for pypa/packaging to lowercase the name? There's
a comment here
<https://github.com/pypa/packaging/blob/16.8/packaging/requirements.py#L86>
about normalizing the requirement's name, which might be what this is
referring to.

kvikshaug added a commit to DD-DeCaF/iam that referenced this issue May 16, 2018

jfly added a commit to jfly/pipenv that referenced this issue May 19, 2018

When formatting a requirement, only lowercase its name.
This fixes pypa#2113.

This bug was introduced in
<jazzband/pip-tools#452> as a band-aid fix to
<jazzband/pip-tools#431>. Pipenv then copied
that code in <pypa@2553ebc#diff-b56b95ccea8595a0f6f24ea753842976>, and inherited this latent bug.

Maybe the right fix is for pypa/packaging to lowercase the name? There's
a comment here
<https://github.com/pypa/packaging/blob/16.8/packaging/requirements.py#L86>
about normalizing the requirement's name, which might be what this is
referring to.

jfly added a commit to jfly/pipenv that referenced this issue May 23, 2018

When formatting a requirement, only lowercase its name.
This fixes pypa#2113.

This bug was introduced in
<jazzband/pip-tools#452> as a band-aid fix to
<jazzband/pip-tools#431>. Pipenv then copied
that code in <pypa@2553ebc#diff-b56b95ccea8595a0f6f24ea753842976>, and inherited this latent bug.

Maybe the right fix is for pypa/packaging to lowercase the name? There's
a comment here
<https://github.com/pypa/packaging/blob/16.8/packaging/requirements.py#L86>
about normalizing the requirement's name, which might be what this is
referring to.

jfly added a commit to jfly/pipenv that referenced this issue May 23, 2018

When formatting a requirement, only lowercase its name.
This fixes pypa#2113.

This bug was introduced in
<jazzband/pip-tools#452> as a band-aid fix to
<jazzband/pip-tools#431>. Pipenv then copied
that code in <pypa@2553ebc#diff-b56b95ccea8595a0f6f24ea753842976>, and inherited this latent bug.

Maybe the right fix is for pypa/packaging to lowercase the name? There's
a comment here
<https://github.com/pypa/packaging/blob/16.8/packaging/requirements.py#L86>
about normalizing the requirement's name, which might be what this is
referring to.

To test this, I invented a new, very simple python package called
`depends-on-marked-package`. The setup.py for this package is just:

```python
import setuptools

setuptools.setup(
    name="depends-on-marked-package",
    version="0.0.1",
    packages=setuptools.find_packages(),
    install_requires=['pytz; platform_python_implementation=="CPython"'],
)
```

This is a simplified version of gevent's setup.py's install_requires upon greenlet.
@Bogdanp

This comment has been minimized.

Bogdanp commented May 24, 2018

Just an FYI: this should probably not have been closed yet since the fix hasn't been released (we just got bit by it).

@Turbo87

This comment has been minimized.

Turbo87 commented May 24, 2018

@Bogdanp it's fixed on master though and keeping issues open until the fix is released puts quite a bit more workload on the maintainers.

@Bogdanp

This comment has been minimized.

Bogdanp commented May 24, 2018

@Turbo87 if that's the project's policy, then fair enough. I thought it was worth pointing out since GitHub most likely closed this automatically. That said, the fact that it adds some maintenance burden doesn't mean it's not the right thing to do. :)

@techalchemy

This comment has been minimized.

Member

techalchemy commented May 24, 2018

We try our best not to fight too hard against the tools we are using, none of us is doing this full time and this tracker is more for our own organization than for making announcements or anything like that.

@uranusjr

This comment has been minimized.

Member

uranusjr commented May 25, 2018

Yeah, GitHub is better suited for the current workflow (closing on resolution, not release). Maybe we can add a tag to all un-released issues, but that would still require quite some administration overhead. I wish we have an ops maintainer to handle these kind of things. You don’t even need to know how to code, but alas people are better at pointing out issues than actually trying to solve them 🙂

derpferd added a commit to UMDLARS/tron that referenced this issue May 31, 2018

Hack
Hacked Pipfile.lock to fix: pypa/pipenv#2113
@judgeaxl

This comment has been minimized.

judgeaxl commented Jun 4, 2018

Is there an ETA for when a version containing this fix might be released?

@uranusjr

This comment has been minimized.

Member

uranusjr commented Jun 4, 2018

@judgeaxl From what I’ve heard a version would be out this week.

@techalchemy

This comment has been minimized.

Member

techalchemy commented Jun 4, 2018

Depending on the ongoing merges etc going smoothly it should be in the next day or two

@bigunyak

This comment has been minimized.

bigunyak commented Jun 12, 2018

Any news on when the next release happens? Would be nice to get this fix finally available.

@uranusjr

This comment has been minimized.

Member

uranusjr commented Jun 12, 2018

Track https://github.com/pypa/pipenv/projects/2 to find out! We still have a few pressing bugs need fixing.

@Turbo87

This comment has been minimized.

Turbo87 commented Jun 12, 2018

@uranusjr would it not be possible to release a version with the existing bugfixes instead of waiting for all bugs to be fixed?

@uranusjr

This comment has been minimized.

Member

uranusjr commented Jun 12, 2018

@Turbo87 It’s not that simple (unfortunately). Some bugs are introduced by the bugfix. The release will fix your problem, and introduce problems to others :(

@Turbo87

This comment has been minimized.

Turbo87 commented Jun 12, 2018

I see, thanks for the explanation!

kierenbeckett added a commit to ONSdigital/eq-survey-runner that referenced this issue Jun 19, 2018

kierenbeckett added a commit to ONSdigital/eq-survey-runner that referenced this issue Jun 20, 2018

kierenbeckett added a commit to ONSdigital/eq-survey-runner that referenced this issue Jun 20, 2018

kierenbeckett added a commit to ONSdigital/eq-survey-runner that referenced this issue Jun 22, 2018

kierenbeckett added a commit to ONSdigital/eq-survey-runner that referenced this issue Jun 22, 2018

kierenbeckett added a commit to ONSdigital/eq-survey-runner that referenced this issue Jun 22, 2018

PurpleBooth added a commit to ONSdigital/ras-party that referenced this issue Aug 23, 2018

Introduce workaround for pypa/pipenv#2113
Currently we can't cf push if we update greenlets. This will force
the right platform to be used.

PurpleBooth added a commit to ONSdigital/ras-party that referenced this issue Aug 23, 2018

Zipkin and pypa/pipenv#2113 (#165)
* Revert "Revert "Zipkin flask (#161)" (#164)"

This reverts commit 65422ec.

* Introduce workaround for pypa/pipenv#2113

Currently we can't cf push if we update greenlets. This will force
the right platform to be used.

PurpleBooth added a commit to ONSdigital/ras-collection-instrument that referenced this issue Aug 23, 2018

Add pypa/pipenv#2113 workaround
Currently the build is failing after an update to gevent. This will
specify the platform and fix the build.

PurpleBooth added a commit to ONSdigital/ras-collection-instrument that referenced this issue Aug 23, 2018

Zipkin + pypa/pipenv#2113 fix (#130)
* Revert "Revert "ZipKin Tracing (#126)" (#129)"

This reverts commit 6304a00.

* Add pypa/pipenv#2113 workaround

Currently the build is failing after an update to gevent. This will
specify the platform and fix the build.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment