gevent==20.6.2 should require a specific version of greenlets

[dlamblin]

• gevent version: 20.6.2
• greenlets version 0.4.17
• Python version: cPython 3.7.0 downloaded from docker python:3.7.0-slim-stretch
• Operating System: debian stretch

Description:

My Apache Airflow FlaskAppBuilder webserver installed and ran fine 2h ago with greenlets version v0.4.16;
However making a config change to my dockerfile triggered a rebuild with reinstall which didn't pin the greenlet and brought in the 50 minute old v0.4.17. This logged a complaint about binary incompatibility and didn't serve any pages.

/usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
[2020-09-22 11:43:24 +0000] [628] [INFO] Starting gunicorn 20.0.4
[2020-09-22 11:43:24 +0000] [628] [INFO] Listening at: http://0.0.0.0:8443 (628)
[2020-09-22 11:43:24 +0000] [628] [INFO] Using worker: gevent
[2020-09-22 11:43:24 +0000] [631] [INFO] Booting worker with pid: 631
[2020-09-22 11:43:24 +0000] [632] [INFO] Booting worker with pid: 632
/usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
[2020-09-22 11:43:24 +0000] [633] [INFO] Booting worker with pid: 633
/usr/local/lib/python3.7/site-packages/gunicorn/workers/ggevent.py:53: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['urllib3.util.ssl_ (/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py)', 'urllib3.util (/usr/local/lib/python3.7/site-packages/urllib3/util/__init__.py)'].
  monkey.patch_all()

Rebuilding with 0.4.16 and everything is fine again.

These constraints (failing to mention greenlets) were used: https://github.com/apache/airflow/blob/constraints-1.10.12/constraints-3.7.txt

Presumably if gevent needs a specific version of greenlets it should pin that version in the released packages.

[jamadden]

Thanks for the report. I understand your frustration. There's very little gevent can do here.

Not all greenlet version changes introduce any incompatibility issues; most just fix bugs or add new platforms so I don't want to introduce an exact version number pin in gevent.

Even if I did pin to exactly one version of greenlet, that's not enough because many of its features, and thus the resulting binary compatibility, can be set at compile time. Building greenlet 0.4.17 with -DGREENLET_USE_CONTEXT_VARS=0, for example, would likely correct this issue for you.

[snaury]

I yanked 0.4.17 until contextvars support could be rewritten in a binary compatible manner or issue is worked around in some other way.

[dlamblin]

Thanks to both of you; I do understand that pinning a version isn't common (look at the length of that constraints file), but the errors one gets from a dependency of a dependency are difficult to recognize, diagnose and work around for many users that are just trying to install a bigger package … so my opinion is that the python community's practice is not particularly helpful. It's difficult to trust a pip install of apache-airflow if it breaks basically every few weeks due to one of the great many deps. OTOH a package like thrift-sasl pins a version of thrift and goes unmaintained for half decade, and it turns out the warning that it wouldn't work with a newer version of thrift is untrue, leading again to no trust for the advertised in-compatibility. It's a pickle.