Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSL.py recursion crash in (eventlet > 0.17.4) and python 3.6 #371

Open
Renmusxd opened this issue Jan 4, 2017 · 31 comments

Comments

Projects
None yet
@Renmusxd
Copy link

commented Jan 4, 2017

Using python 3.6 with the following code causes an infinite recursion of python super calls leading to a crash

import eventlet
eventlet.monkey_patch()

import os
import socket
from flask import Flask, render_template, request, Response, send_file
from flask import make_response
from flask_socketio import SocketIO
from OpenSSL import SSL, crypto

'''
requirements.txt
cffi==1.9.1
click==6.6
cryptography==1.7.1
enum-compat==0.0.2
eventlet==0.20.1
Flask==0.12
Flask-SocketIO==2.8.2
greenlet==0.4.11
idna==2.2
itsdangerous==0.24
Jinja2==2.8.1
MarkupSafe==0.23
pyasn1==0.1.9
pycparser==2.17
pyOpenSSL==16.2.0
python-engineio==1.1.0
python-socketio==1.6.2
six==1.10.0
Werkzeug==0.11.15
'''

app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'
async_mode = None
socketio = SocketIO(app, async_mode=async_mode)
def create_self_signed_cert(certfile, keyfile, certargs, cert_dir="."):
    C_F = os.path.join(cert_dir, certfile)
    K_F = os.path.join(cert_dir, keyfile)
    if not os.path.exists(C_F) or not os.path.exists(K_F):
        k = crypto.PKey()
        k.generate_key(crypto.TYPE_RSA, 1024)
        cert = crypto.X509()
        cert.get_subject().C = certargs["Country"]
        cert.get_subject().ST = certargs["State"]
        cert.get_subject().L = certargs["City"]
        cert.get_subject().O = certargs["Organization"]
        cert.get_subject().OU = certargs["Org. Unit"]
        cert.get_subject().CN = 'Example'
        cert.set_serial_number(1000)
        cert.gmtime_adj_notBefore(0)
        cert.gmtime_adj_notAfter(315360000)
        cert.set_issuer(cert.get_subject())
        cert.set_pubkey(k)
        cert.sign(k, 'sha1')
        open(C_F, "wb").write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
        open(K_F, "wb").write(crypto.dump_privatekey(crypto.FILETYPE_PEM, k))

CERT_FILE = "cert.pem"
KEY_FILE = "key.pem"
create_self_signed_cert(CERT_FILE, KEY_FILE,
                            certargs=
                            {"Country": "US",
                             "State": "NY",
                             "City": "Ithaca",
                             "Organization": "Python-Bugs",
                             "Org. Unit": "Proof of Concept"})
socketio.run(app, debug=True, use_reloader=False, certfile=CERT_FILE, keyfile=KEY_FILE, port=5500)

Trace:

Traceback (most recent call last):
File "[...]/server.py", line 231, in
socketio.run(app, debug=True, use_reloader=False, certfile=CERT_FILE, keyfile=KEY_FILE, port=5500)
File "[...]/.virtualenvs/bot/lib/python3.6/site-packages/flask_socketio/init.py", line 493, in run
run_server()
File "[...]/.virtualenvs/bot/lib/python3.6/site-packages/flask_socketio/init.py", line 485, in run_server
**ssl_params)
File "[...]/.virtualenvs/bot/lib/python3.6/site-packages/eventlet/convenience.py", line 126, in wrap_ssl
return wrap_ssl_impl(sock, *a, **kw)
File "[...]/.virtualenvs/bot/lib/python3.6/site-packages/eventlet/green/ssl.py", line 379, in wrap_socket
return GreenSSLSocket(sock, *a, **kw)
File "[...]/.virtualenvs/bot/lib/python3.6/site-packages/eventlet/green/ssl.py", line 68, in init
ca_certs, do_handshake_on_connect and six.PY2, *args, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 737, in init
self._context.verify_mode = cert_reqs
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 479, in verify_mode
super(SSLContext, SSLContext).verify_mode.set(self, value)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 479, in verify_mode
super(SSLContext, SSLContext).verify_mode.set(self, value)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 479, in verify_mode
super(SSLContext, SSLContext).verify_mode.set(self, value)
[Previous line repeated 325 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

Issue also posted to http://bugs.python.org/issue29149, and miguelgrinberg/Flask-SocketIO#193, but since it's eventlet version dependent may also be relevant here.

@Renmusxd

This comment has been minimized.

Copy link
Author

commented Jan 4, 2017

Simpler code to reproduce bug:

#!/usr/bin/env python3

import eventlet
eventlet.monkey_patch()

import sys

from eventlet import wsgi

def hello_world (env, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    return ['Hello, World!\r\n']


def main ():
    listener = eventlet.wrap_ssl(
        eventlet.listen(('127.0.0.1', 8000)),
        certfile='does-not-exist',
        keyfile='does-not-exist',
        server_side=True)

    wsgi.server(listener, hello_world)

if __name__ == '__main__':
    sys.exit(main())

Shamelessly taken from: https://www.bountysource.com/issues/32684217-using-both-eventlet-monkey_patch-and-eventlet-wsgi-server-together-with-ssl-fails-with-ssl-sslwantreaderror, but now with different error

temoto added a commit that referenced this issue Jan 5, 2017

mythmon added a commit to mythmon/normandy that referenced this issue Feb 16, 2017

Downgrade eventlet to 0.17.4 to fix Python 3.6 compatibility
Eventlet > 0.17.4 has a problem with Python 3.6: eventlet/eventlet#371

mythmon added a commit to mythmon/normandy that referenced this issue Feb 16, 2017

Downgrade eventlet to 0.17.4 to fix Python 3.6 compatibility
Eventlet > 0.17.4 has a problem with Python 3.6: eventlet/eventlet#371
@Hoffs

This comment has been minimized.

Copy link

commented Feb 21, 2017

Similar error while using Requests library, fixed by reverting to 0.17.4 as mentioned.:

2017-02-21T16:56:44.729121+00:00 app[worker1.1]: [2017-02-21 16:56:44,726: ERROR/MainProcess] Task twitch_stats.tasks.get_stats_by_id[9e7eb62b-3fbf-47c9-ab0c-6e02f078bb58] raised unexpected: RecursionError('maximum recursion depth exceeded while calling a Python object',)
2017-02-21T16:56:44.729124+00:00 app[worker1.1]: Traceback (most recent call last):
2017-02-21T16:56:44.729125+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/celery/app/trace.py", line 367, in trace_task
2017-02-21T16:56:44.729126+00:00 app[worker1.1]:     R = retval = fun(*args, **kwargs)
2017-02-21T16:56:44.729126+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/celery/app/trace.py", line 622, in __protected_call__
2017-02-21T16:56:44.729127+00:00 app[worker1.1]:     return self.run(*args, **kwargs)
2017-02-21T16:56:44.729128+00:00 app[worker1.1]:   File "/app/twitch_stats/tasks.py", line 34, in get_stats_by_id
2017-02-21T16:56:44.729129+00:00 app[worker1.1]:     TwitchStats.objects.get_stats(twitch_id=tid)
2017-02-21T16:56:44.729130+00:00 app[worker1.1]:   File "/app/twitch_stats/managers.py", line 160, in get_stats
2017-02-21T16:56:44.729130+00:00 app[worker1.1]:     r = requests.get(url=url, headers=headers)
2017-02-21T16:56:44.729132+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/api.py", line 70, in get
2017-02-21T16:56:44.729132+00:00 app[worker1.1]:     return request('get', url, params=params, **kwargs)
2017-02-21T16:56:44.729133+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/api.py", line 56, in request
2017-02-21T16:56:44.729134+00:00 app[worker1.1]:     return session.request(method=method, url=url, **kwargs)
2017-02-21T16:56:44.729134+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
2017-02-21T16:56:44.729135+00:00 app[worker1.1]:     resp = self.send(prep, **send_kwargs)
2017-02-21T16:56:44.729136+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
2017-02-21T16:56:44.729137+00:00 app[worker1.1]:     r = adapter.send(request, **kwargs)
2017-02-21T16:56:44.729137+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
2017-02-21T16:56:44.729138+00:00 app[worker1.1]:     timeout=timeout
2017-02-21T16:56:44.729139+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
2017-02-21T16:56:44.729140+00:00 app[worker1.1]:     chunked=chunked)
2017-02-21T16:56:44.729141+00:00 app[worker1.1]:     self._validate_conn(conn)
2017-02-21T16:56:44.729141+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
2017-02-21T16:56:44.729142+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 844, in _validate_conn
2017-02-21T16:56:44.729143+00:00 app[worker1.1]:     conn.connect()
2017-02-21T16:56:44.729143+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 314, in connect
2017-02-21T16:56:44.729144+00:00 app[worker1.1]:     cert_reqs=resolve_cert_reqs(self.cert_reqs),
2017-02-21T16:56:44.729145+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 264, in create_urllib3_context
2017-02-21T16:56:44.729146+00:00 app[worker1.1]:     context.options |= options
2017-02-21T16:56:44.729146+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/ssl.py", line 459, in options
2017-02-21T16:56:44.729148+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/ssl.py", line 459, in options
2017-02-21T16:56:44.729147+00:00 app[worker1.1]:     super(SSLContext, SSLContext).options.__set__(self, value)
2017-02-21T16:56:44.729149+00:00 app[worker1.1]:   File "/app/.heroku/python/lib/python3.6/ssl.py", line 459, in options
2017-02-21T16:56:44.729148+00:00 app[worker1.1]:     super(SSLContext, SSLContext).options.__set__(self, value)
2017-02-21T16:56:44.729149+00:00 app[worker1.1]:     super(SSLContext, SSLContext).options.__set__(self, value)
2017-02-21T16:56:44.729150+00:00 app[worker1.1]:   [Previous line repeated 109 more times]
2017-02-21T16:56:44.729151+00:00 app[worker1.1]:   File "...", line -1, in [rest of traceback truncated]
2017-02-21T16:56:44.729200+00:00 app[worker1.1]: RecursionError: maximum recursion depth exceeded while calling a Python object
@j-walker23

This comment has been minimized.

Copy link

commented Mar 17, 2017

Hi all, what is needed to get new eventlet versions working with python 3.6, either on the python side or eventlet side?

@temoto

This comment has been minimized.

Copy link
Member

commented Mar 17, 2017

Try to uninstall pyopenssl if possible, please say if it helps.

@temoto

This comment has been minimized.

Copy link
Member

commented Mar 21, 2017

@j-walker23 @Hoffs- @Renmusxd please try pip uninstall pyopenssl and tell if it helps.

@Heanthor

This comment has been minimized.

Copy link

commented Mar 22, 2017

I'm having this same problem as well, but PyOpenSSL is not installed. Also using Flask 0.12, Flask-SocketIO 2.8.6.

@temoto

This comment has been minimized.

Copy link
Member

commented Mar 22, 2017

@Heanthor thank you.

@j-walker23

This comment has been minimized.

Copy link

commented Mar 25, 2017

Thanks for the replies! I will test on my gce box running ubuntu and report back.

@j-walker23

This comment has been minimized.

Copy link

commented Mar 25, 2017

@temoto I actually didn't have pyopenssl installed either. Should I only be trying eventlet 0.17.4?

@temoto

This comment has been minimized.

Copy link
Member

commented Mar 25, 2017

@j-walker23 I'm sorry old version is the only workaround right now.

I understand this is an important issue but haven't find solution yet.

@j-walker23

This comment has been minimized.

Copy link

commented Mar 26, 2017

@temoto no problem at all. just was curious because i seem to be able to run the latest eventlet on my mac with py36. but when deployed on ubuntu it has this error so i assumed i was just screwing it up. Thank you, really appreciate the lib!

@justdoit0823

This comment has been minimized.

Copy link

commented Mar 28, 2017

The ssl module of Python3.6 has made variable options, verify_flags, verify_mode of SSLContext as property, so setting value of the specified property caused function recursively call after eventlet monkey patch. The following code can be found in Python3.6 ssl module.

@property
def options(self):
    return Options(super().options)

@options.setter
def options(self, value):
    super(SSLContext, SSLContext).options.__set__(self, value)

@property
def verify_flags(self):
    return VerifyFlags(super().verify_flags)

@verify_flags.setter
def verify_flags(self, value):
    super(SSLContext, SSLContext).verify_flags.__set__(self, value)

@property
def verify_mode(self):
    value = super().verify_mode
    try:
        return VerifyMode(value)
    except ValueError:
        return value

@verify_mode.setter
def verify_mode(self, value):
    super(SSLContext, SSLContext).verify_mode.__set__(self, value)

As gevent's fixup of this issue:

if hasattr(orig_SSLContext.options, 'setter'):
        # In 3.6, these became properties. They want to access the
        # property __set__ method in the superclass, and they do so by using
        # super(SSLContext, SSLContext). But we rebind SSLContext when we monkey
        # patch, which causes infinite recursion.
        # https://github.com/python/cpython/commit/328067c468f82e4ec1b5c510a4e84509e010f296
        @orig_SSLContext.options.setter
        def options(self, value):
            super(orig_SSLContext, orig_SSLContext).options.__set__(self, value)

        @orig_SSLContext.verify_flags.setter
        def verify_flags(self, value):
            super(orig_SSLContext, orig_SSLContext).verify_flags.__set__(self, value)

        @orig_SSLContext.verify_mode.setter
        def verify_mode(self, value):
            super(orig_SSLContext, orig_SSLContext).verify_mode.__set__(self, value)

Insert if statement in SSLContext definition and decorate set property function.

@temoto

This comment has been minimized.

Copy link
Member

commented Mar 29, 2017

@justdoit0823 thanks a lot!

temoto added a commit that referenced this issue Apr 4, 2017

@temoto

This comment has been minimized.

Copy link
Member

commented Apr 4, 2017

Everybody, please, try this version.

pip install https://github.com/eventlet/eventlet/archive/de06878e5a295bfbbddca0048c3453d16168a676.zip

@temoto

This comment has been minimized.

Copy link
Member

commented Apr 5, 2017

@richardasaurus that's very strange, please show full terminal session how you got it.

@temoto

This comment has been minimized.

Copy link
Member

commented Apr 5, 2017

Hey guys, the fix was released on PyPI as v0.21.0. Please tell me if it helps.

@j-walker23

This comment has been minimized.

Copy link

commented Apr 5, 2017

Awesome, thank you!

@temoto

This comment has been minimized.

Copy link
Member

commented Apr 7, 2017

@richardasaurus well show everything related to this error from deployment. I can only help knowing how to reproduce the problem.

@kvikshaug kvikshaug referenced this issue Apr 20, 2017

Closed

Upgrade to Python 3.6 #236

1 of 1 task complete
@daviskirk

This comment has been minimized.

Copy link

commented May 9, 2017

I can confirm that 0.21.0 fixes the eventlet / requests issue. Can't seem to reproduce the compile error @richardasaurus has.

@benkuhn

This comment has been minimized.

Copy link

commented May 11, 2017

I'm still seeing a similar error on eventlet 0.21.0 due to some kind of incredibly bizarre interaction with logging to the root logger at toplevel: https://github.com/benkuhn/eventlet-repro

@temoto

This comment has been minimized.

Copy link
Member

commented May 11, 2017

@benkuhn thank you very much for reproduction repo. Please post python -V.

@benkuhn

This comment has been minimized.

Copy link

commented May 11, 2017

$ python -V
Python 3.6.1

I've pushed an even more minimal repro. Still working on minimizing further. Note that it may be related to newrelic vendoring an old urllib version.

@ahartoto

This comment has been minimized.

Copy link

commented Jun 9, 2017

Hi, I see this error when I used boto3 functionality. Do we have any fix that I can try out?

@temoto

This comment has been minimized.

Copy link
Member

commented Jun 10, 2017

@ahartoto not yet.

@richmondwang

This comment has been minimized.

Copy link

commented Jun 30, 2017

Any workaround for now? I cant use 0.17.4 because I get sqlalchemy.orm.exc.DetachedInstanceError

@temoto

This comment has been minimized.

Copy link
Member

commented Jun 30, 2017

@ahartoto @richmondwang please try pip install -U 'newrelic>=2.86.3.70' worked for me

@richmondwang

This comment has been minimized.

Copy link

commented Jul 3, 2017

I did pip install -U 'newrelic>=2.86.3.70' but still the same. I use python36, and with same requirements as OP's.

@temoto

This comment has been minimized.

Copy link
Member

commented Jul 3, 2017

@richmondwang can you provide reproduce instruction?

@dhhagan

This comment has been minimized.

Copy link

commented Dec 7, 2017

Is there an update/recommendation on how to bypass this issue with the most up-to-date versions of eventlet and python3.6.3?

@dhhagan

This comment has been minimized.

Copy link

commented Dec 7, 2017

@richardasaurus Ahh thanks. It appears it was an issue with where I used monkey_patch(), which I moved to the very top of my project file after reading this best practices list.

@xuhuigithub

This comment has been minimized.

Copy link

commented Jun 6, 2018

Hi, everybody, I found the resolution there.gevent/gevent#941

openstack-gerrit pushed a commit to openstack/nova that referenced this issue Mar 22, 2019

Eventlet monkey patching should be as early as possible
We were seeing infinite recursion opening an ssl socket when running
various combinations of python3, eventlet, and urllib3. It is not
clear exactly what combination of versions are affected, but for
background there is an example of this issue documented here:

eventlet/eventlet#371

The immediate cause in nova's case was that we were calling
eventlet.monkey_patch() after importing urllib3. Specifically, change
Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the
nova.utils.monkey_patch() method to make monkey patching common
between WSGI and non-WSGI services. Unfortunately, before executing
this method you must first import nova.utils, which imports a large
number of modules itself. Anything imported (transitively) by
nova.utils would therefore be imported before monkey patching, which
included urllib3. This triggers the infinite recursion problem
described above if you have an affected combination of library
versions.

While this specific issue may eventually be worked around or fixed in
eventlet or urllib3, it remains true that eventlet best practises are
to monkey patch as early as possible, which we were not doing. To
avoid this and hopefully future similar issues, this change ensures
that monkey patching happens as early as possible, and only a minimum
number of modules are imported first.

This change fixes monkey patching for both non-wsgi and wsgi callers:

* Non-WSGI services (nova/cmd)

  This is fixed by using the new monkey_patch module, which has minimal
  dependencies.

* WSGI services (nova/api/openstack)

  This is fixed both by using the new monkey_patch module, and by moving
  the patching point up one level so that it is done before importing
  anything in nova/api/openstack/__init__.py.

  This move causes issues for some external tools which load this path
  from nova and now monkey patch where they previously did not. However,
  it is unfortunately unavoidable to enable monkey patching for the wsgi
  entry point without major restructuring. This change includes a
  workaround for sphinx to avoid this issue.

This change has been through several iterations. I started with what
seemed like the simplest and most obvious change, and moved on as I
discovered more interactions which broke. It is clear that eventlet
monkey patching is extremely fragile, especially when done implicitly at
module load time as we do. I would advocate a code restructure to
improve this situation, but I think the time would be better spent
removing the eventlet dependency entirely.

Co-authored-by: Lee Yarwood <lyarwood@redhat.com>

Closes-Bug: #1808975
Closes-Bug: #1808951
Change-Id: Id46e76666b553a10ec4654d4418a9884975b5b95

openstack-gerrit added a commit to openstack/openstack that referenced this issue Mar 22, 2019

Update git submodules
* Update nova from branch 'master'
  - Merge "Eventlet monkey patching should be as early as possible"
  - Eventlet monkey patching should be as early as possible
    
    We were seeing infinite recursion opening an ssl socket when running
    various combinations of python3, eventlet, and urllib3. It is not
    clear exactly what combination of versions are affected, but for
    background there is an example of this issue documented here:
    
    eventlet/eventlet#371
    
    The immediate cause in nova's case was that we were calling
    eventlet.monkey_patch() after importing urllib3. Specifically, change
    Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the
    nova.utils.monkey_patch() method to make monkey patching common
    between WSGI and non-WSGI services. Unfortunately, before executing
    this method you must first import nova.utils, which imports a large
    number of modules itself. Anything imported (transitively) by
    nova.utils would therefore be imported before monkey patching, which
    included urllib3. This triggers the infinite recursion problem
    described above if you have an affected combination of library
    versions.
    
    While this specific issue may eventually be worked around or fixed in
    eventlet or urllib3, it remains true that eventlet best practises are
    to monkey patch as early as possible, which we were not doing. To
    avoid this and hopefully future similar issues, this change ensures
    that monkey patching happens as early as possible, and only a minimum
    number of modules are imported first.
    
    This change fixes monkey patching for both non-wsgi and wsgi callers:
    
    * Non-WSGI services (nova/cmd)
    
      This is fixed by using the new monkey_patch module, which has minimal
      dependencies.
    
    * WSGI services (nova/api/openstack)
    
      This is fixed both by using the new monkey_patch module, and by moving
      the patching point up one level so that it is done before importing
      anything in nova/api/openstack/__init__.py.
    
      This move causes issues for some external tools which load this path
      from nova and now monkey patch where they previously did not. However,
      it is unfortunately unavoidable to enable monkey patching for the wsgi
      entry point without major restructuring. This change includes a
      workaround for sphinx to avoid this issue.
    
    This change has been through several iterations. I started with what
    seemed like the simplest and most obvious change, and moved on as I
    discovered more interactions which broke. It is clear that eventlet
    monkey patching is extremely fragile, especially when done implicitly at
    module load time as we do. I would advocate a code restructure to
    improve this situation, but I think the time would be better spent
    removing the eventlet dependency entirely.
    
    Co-authored-by: Lee Yarwood <lyarwood@redhat.com>
    
    Closes-Bug: #1808975
    Closes-Bug: #1808951
    Change-Id: Id46e76666b553a10ec4654d4418a9884975b5b95

openstack-gerrit added a commit to openstack/openstack that referenced this issue May 3, 2019

Update git submodules
* Update cinder from branch 'master'
  - Merge "Raise eventlet lower-constraint to 0.22.0"
  - Raise eventlet lower-constraint to 0.22.0
    
    Eventlet has a known issue when running under Python 3.6 [1]. This was
    noticed in another OpenStack repo. As far as I know, we have not hit
    this in Cinder. But since many drivers use requests and most are not
    running third party CI under 3.6 yet, this is an attempt to avoid
    running into an issue with the versions of eventlet that have this
    issue.
    
    [1] eventlet/eventlet#371
    
    Change-Id: Ia6139c052091eaee6b49a55f57385e650508a891
    Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>

openstack-gerrit pushed a commit to openstack/cinder that referenced this issue May 3, 2019

Raise eventlet lower-constraint to 0.22.0
Eventlet has a known issue when running under Python 3.6 [1]. This was
noticed in another OpenStack repo. As far as I know, we have not hit
this in Cinder. But since many drivers use requests and most are not
running third party CI under 3.6 yet, this is an attempt to avoid
running into an issue with the versions of eventlet that have this
issue.

[1] eventlet/eventlet#371

Change-Id: Ia6139c052091eaee6b49a55f57385e650508a891
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>

openstack-gerrit pushed a commit to openstack/nova that referenced this issue May 4, 2019

Eventlet monkey patching should be as early as possible
We were seeing infinite recursion opening an ssl socket when running
various combinations of python3, eventlet, and urllib3. It is not
clear exactly what combination of versions are affected, but for
background there is an example of this issue documented here:

eventlet/eventlet#371

The immediate cause in nova's case was that we were calling
eventlet.monkey_patch() after importing urllib3. Specifically, change
Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the
nova.utils.monkey_patch() method to make monkey patching common
between WSGI and non-WSGI services. Unfortunately, before executing
this method you must first import nova.utils, which imports a large
number of modules itself. Anything imported (transitively) by
nova.utils would therefore be imported before monkey patching, which
included urllib3. This triggers the infinite recursion problem
described above if you have an affected combination of library
versions.

While this specific issue may eventually be worked around or fixed in
eventlet or urllib3, it remains true that eventlet best practises are
to monkey patch as early as possible, which we were not doing. To
avoid this and hopefully future similar issues, this change ensures
that monkey patching happens as early as possible, and only a minimum
number of modules are imported first.

This change fixes monkey patching for both non-wsgi and wsgi callers:

* Non-WSGI services (nova/cmd)

  This is fixed by using the new monkey_patch module, which has minimal
  dependencies.

* WSGI services (nova/api/openstack)

  This is fixed both by using the new monkey_patch module, and by moving
  the patching point up one level so that it is done before importing
  anything in nova/api/openstack/__init__.py.

  This move causes issues for some external tools which load this path
  from nova and now monkey patch where they previously did not. However,
  it is unfortunately unavoidable to enable monkey patching for the wsgi
  entry point without major restructuring. This change includes a
  workaround for sphinx to avoid this issue.

This change has been through several iterations. I started with what
seemed like the simplest and most obvious change, and moved on as I
discovered more interactions which broke. It is clear that eventlet
monkey patching is extremely fragile, especially when done implicitly at
module load time as we do. I would advocate a code restructure to
improve this situation, but I think the time would be better spent
removing the eventlet dependency entirely.

Co-authored-by: Lee Yarwood <lyarwood@redhat.com>

Closes-Bug: #1808975
Closes-Bug: #1808951
Change-Id: Id46e76666b553a10ec4654d4418a9884975b5b95
(cherry picked from commit 3c5e2b0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.