Cannot install gevent 1.3.0 on CentOS 7.5

[apmarathe]

• gevent version: 1.3.0
• Python version: 3.4.2
• Operating System: CentOS 7.5

Description:

Cannot install gevent 1.3.0 on CentOS 7.5

Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/build/gevent/setup.py", line 406, in <module>
        run_setup(EXT_MODULES, run_make=_BUILDING)
      File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/build/gevent/setup.py", line 390, in run_setup
        "signal_os_incompat = gevent.monkey:_subscribe_signal_os",
      File "/opt/rh/rh-python34/root/usr/lib64/python3.4/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/setuptools/dist.py", line 260, in __init__
        self.fetch_build_eggs(attrs.pop('setup_requires'))
      File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/setuptools/dist.py", line 285, in fetch_build_eggs
        replace_conflicting=True
      File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/pkg_resources.py", line 603, in resolve
        requirements = list(requirements)[::-1]
      File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/pkg_resources.py", line 2678, in parse_requirements
        "version spec")
      File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/pkg_resources.py", line 2656, in scan_list
        raise ValueError(msg, line, "at", line[p:])
    ValueError: ("Expected ',' or end-of-list in", "cffi >= 1.11.5 ; sys_platform == 'win32' and platform_python_implementation == 'CPython'", 'at', " ; sys_platform == 'win32' and platform_python_implementation == 'CPython'")
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/build/gevent/setup.py", line 406, in <module>

    run_setup(EXT_MODULES, run_make=_BUILDING)

  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/build/gevent/setup.py", line 390, in run_setup

    "signal_os_incompat = gevent.monkey:_subscribe_signal_os",

  File "/opt/rh/rh-python34/root/usr/lib64/python3.4/distutils/core.py", line 108, in setup

    _setup_distribution = dist = klass(attrs)

  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/setuptools/dist.py", line 260, in __init__

    self.fetch_build_eggs(attrs.pop('setup_requires'))

  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/setuptools/dist.py", line 285, in fetch_build_eggs

    replace_conflicting=True

  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/pkg_resources.py", line 603, in resolve

    requirements = list(requirements)[::-1]

  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/pkg_resources.py", line 2678, in parse_requirements

    "version spec")

  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/pkg_resources.py", line 2656, in scan_list

    raise ValueError(msg, line, "at", line[p:])

ValueError: ("Expected ',' or end-of-list in", "cffi >= 1.11.5 ; sys_platform == 'win32' and platform_python_implementation == 'CPython'", 'at', " ; sys_platform == 'win32' and platform_python_implementation == 'CPython'")

What I've run:

Tried to install gevent in a freshly provisioned CentOS 7.5 box.

[jamadden]

Thanks for the report. Your version of setuptools is too old to understand environment markers (and presumably wheels). You'll need to update it.

[apmarathe]

Hello,

Just tried to install with setuptools version 39.1.0, and it still fails with the same error message.

[jamadden]

The wheels are built on CentOS 5 and I can't reproduce that problem there.

It might help if you showed the complete sequence of commands you use and their output, starting from when you create the virtual environment.

[apmarathe]

We have a vagrant CentOS 7.5 box.

First, we enable virtualenv-3.4 (the python version is 3.4.2)
Then, in an activated virtual environment, we issue:

pip install -r file.txt

Inside file.txt are the following lines (only a few lines shown):

sqlalchemy==1.2.7
pyodbc==4.0.23
requests==2.18.4
Babel==2.5.3
grequests==0.3.0

The last package, grequests, is what installs gevent 1.3.0.

Interestingly, if we add an older version of gevent explicitly, like so:

sqlalchemy==1.2.7
pyodbc==4.0.23
requests==2.18.4
Babel==2.5.3
gevent==1.2.2
grequests==0.3.0

then the installation problem disappears.

[jamadden]

I would suggest that the first thing you do in your virtual environment is pip install -U setuptools. (If you do that as part of -r file.txt it may be too late.)

EDIT: FWIW, Python 3.4.4 ships setuptools-18.2 in its -m venv, which is far too old. Presumably Python 3.4.2 is at least that old. 3.4.8 has setuptools 28.8, which should work.

[apmarathe]

Thanks, your suggestion helped.

Now we are getting a greenlet-related error. This one is from a unit-test running, and not during installation:

    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/src/reports/r1035_bank_statement/__init__.py", line 3, in <module>
    from reports.r1035_bank_statement.r1035_bank_statement import (
  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/src/reports/r1035_bank_statement/r1035_bank_statement.py", line 7, in <module>
    import gevent.monkey
  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/gevent/__init__.py", line 87, in <module>
    from gevent._hub_local import get_hub
  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/gevent/_hub_local.py", line 101, in <module>
    import_c_accel(globals(), 'gevent.__hub_local')
  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib/python3.4/site-packages/gevent/_util.py", line 105, in import_c_accel
    mod = importlib.import_module(cname)
  File "/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/lib64/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "src/gevent/__greenlet_primitives.pxd", line 12, in init gevent.__hub_local
ImportError: No module named 'greenlet'

[jamadden]

Look closely at the output of pip install to see why greenlet was not installed. It is a dependency of gevent, but it uses environment markers also. Certain broken tools fail to handle that correctly (e.g., #1207). Perhaps your pip is also outdated?

[apmarathe]

I think that was it.

In an activated environment, we now do:

pip install -U pip
pip install -U setuptools

The greenlet problem has also disappeared.
It that a correct solution?

[apmarathe]

By the way, one test case is still failing with the following warning:

/vagrant/CurrentRelease/PacWestServices/dms/dms_reporting_service/src/reports/r1035_bank_statement/r1035_bank_statement.py:8: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016
  gevent.monkey.patch_all(thread=False, select=False)

We do do monkey-patching as the first statement of the relevant file.
Another observation: this test case created two image files, and their sizes are off by 1 byte each. One file is larger by 1 byte, and another is smaller by 1 byte.

[jamadden]

That seems fine to me. Newer point releases of Python include the required versions so you could also consider updating that.

[jamadden]

We do do monkey-patching as the first statement of the relevant file.

Something else in your environment must be importing ssl during startup. Are you in a namespace package established by pkg_resources? (#1108) Could it be in sitecustomize.py? You'll need to trace the imports with -vvv (or insert a breakpoint in ssl.py) to find out. Ideally it would be fixed because when you upgrade to Python 3.5 or 3.6 that early import is liable to cause broken things to happen.

[apmarathe]

This is during the running of a test case using nosetests, so I am not sure how to use the -vvv option with that. Also, I am running just a single test case, not a suite of them, and the test case file does not contain any import of ssl. It's possible that ssl is imported by one of the other packages? I don't think we use sitecustomize.py.

[jamadden]

Yes, it's possible something during test discovery imported SSL. I personally don't recommend monkey-patching at the top of regular unittest files that will be run in the same process by a generic testrunner like nose/nose2/zope.testrunner, because the discovery process will import things in unpredictable orders and you can wind up with bad interactions.

gevent's own test suite forks a new process for each file, guaranteeing no such interactions. (If you have if __name__ == '__main__': unittest.main() in your test file, you can simply use python -m the.module [TheTestClass[.the_test_method]] to run individual tests with much limited discovery.)

In my office job, unit tests simply aren't allowed to monkey-patch---only the main application monkey-patches. If we need to use gevent APIs, we import them and use them directly.

[apmarathe]

Sorry I was not clear. Monkey-patching is only at the top of the code file that the test case is trying to test.

And in that file, the top two lines are:

import gevent.monkey
gevent.monkey.patch_all(thread=False, select=False)

BTW, while on the topic, would you recommend patch_all() without any parameters specified?

[jamadden]

Still, if you're running more than one test in the same process, I wouldn't recommend that. And also still, if you're using a testrunner, it's likely the discovery process is importing something that imports ssl. You'll just have to look into it.

gevent.monkey.patch_all(thread=False, select=False)

BTW, while on the topic, would you recommend patch_all() without any parameters specified?

Yes, I absolutely would recommend a simple call to patch_all(). The defaults are what is tested and recommended. Only in very particular circumstances for very particular needs would I recommend changing them. Hopefully whoever turned those patches off left comments as to why...