Skip to content

Commit 3c5e2b0

Browse files
committed
Eventlet monkey patching should be as early as possible
We were seeing infinite recursion opening an ssl socket when running various combinations of python3, eventlet, and urllib3. It is not clear exactly what combination of versions are affected, but for background there is an example of this issue documented here: eventlet/eventlet#371 The immediate cause in nova's case was that we were calling eventlet.monkey_patch() after importing urllib3. Specifically, change Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the nova.utils.monkey_patch() method to make monkey patching common between WSGI and non-WSGI services. Unfortunately, before executing this method you must first import nova.utils, which imports a large number of modules itself. Anything imported (transitively) by nova.utils would therefore be imported before monkey patching, which included urllib3. This triggers the infinite recursion problem described above if you have an affected combination of library versions. While this specific issue may eventually be worked around or fixed in eventlet or urllib3, it remains true that eventlet best practises are to monkey patch as early as possible, which we were not doing. To avoid this and hopefully future similar issues, this change ensures that monkey patching happens as early as possible, and only a minimum number of modules are imported first. This change fixes monkey patching for both non-wsgi and wsgi callers: * Non-WSGI services (nova/cmd) This is fixed by using the new monkey_patch module, which has minimal dependencies. * WSGI services (nova/api/openstack) This is fixed both by using the new monkey_patch module, and by moving the patching point up one level so that it is done before importing anything in nova/api/openstack/__init__.py. This move causes issues for some external tools which load this path from nova and now monkey patch where they previously did not. However, it is unfortunately unavoidable to enable monkey patching for the wsgi entry point without major restructuring. This change includes a workaround for sphinx to avoid this issue. This change has been through several iterations. I started with what seemed like the simplest and most obvious change, and moved on as I discovered more interactions which broke. It is clear that eventlet monkey patching is extremely fragile, especially when done implicitly at module load time as we do. I would advocate a code restructure to improve this situation, but I think the time would be better spent removing the eventlet dependency entirely. Co-authored-by: Lee Yarwood <lyarwood@redhat.com> Closes-Bug: #1808975 Closes-Bug: #1808951 Change-Id: Id46e76666b553a10ec4654d4418a9884975b5b95
1 parent 59f1f18 commit 3c5e2b0

File tree

9 files changed

+110
-43
lines changed

9 files changed

+110
-43
lines changed

doc/source/conf.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,14 @@
160160
]
161161
# -- Custom extensions --------------------------------------------------------
162162

163+
# NOTE(mdbooth): (2019-03-20) Sphinx loads policies defined in setup.cfg, which
164+
# includes the placement policy at nova/api/openstack/placement/policies.py.
165+
# Loading this imports nova/api/openstack/__init__.py, which imports
166+
# nova.monkey_patch, which will do eventlet monkey patching to the sphinx
167+
# process. As well as being unnecessary and a bad idea, this breaks on
168+
# python3.6 (but not python3.7), so don't do that.
169+
os.environ['OS_NOVA_DISABLE_EVENTLET_PATCHING'] = '1'
170+
163171

164172
def monkey_patch_blockdiag():
165173
"""Monkey patch the blockdiag library.

nova/__init__.py

Lines changed: 0 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -22,14 +22,3 @@
2222
:platform: Unix
2323
:synopsis: Infrastructure-as-a-Service Cloud platform.
2424
"""
25-
26-
import os
27-
28-
os.environ['EVENTLET_NO_GREENDNS'] = 'yes'
29-
30-
# NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet hub
31-
# use a monotonic clock to avoid issues with drifts of system time (see
32-
# LP 1510234 for details)
33-
import oslo_service # noqa
34-
35-
import eventlet # noqa

nova/api/openstack/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
"""
1818
WSGI middleware for OpenStack API controllers.
1919
"""
20+
import nova.monkey_patch # noqa
2021

2122
from oslo_log import log as logging
2223
import routes

nova/api/openstack/wsgi_app.py

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,13 +23,11 @@
2323
from nova import exception
2424
from nova import objects
2525
from nova import service
26-
from nova import utils
2726

2827
CONF = cfg.CONF
2928

3029
CONFIG_FILES = ['api-paste.ini', 'nova.conf']
3130

32-
utils.monkey_patch()
3331
objects.register_all()
3432

3533

nova/cmd/__init__.py

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
1+
# Copyright (c) 2019 Red Hat, Inc.
22
# All Rights Reserved.
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License"); you may
@@ -13,6 +13,4 @@
1313
# License for the specific language governing permissions and limitations
1414
# under the License.
1515

16-
from nova import utils
17-
18-
utils.monkey_patch()
16+
import nova.monkey_patch # noqa

nova/monkey_patch.py

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# Copyright 2010 United States Government as represented by the
2+
# Administrator of the National Aeronautics and Space Administration.
3+
# Copyright 2011 Justin Santa Barbara
4+
# Copyright 2019 Red Hat, Inc.
5+
# All Rights Reserved.
6+
#
7+
# Licensed under the Apache License, Version 2.0 (the "License"); you may
8+
# not use this file except in compliance with the License. You may obtain
9+
# a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing, software
14+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
15+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
16+
# License for the specific language governing permissions and limitations
17+
# under the License.
18+
19+
"""Enable eventlet monkey patching."""
20+
21+
import os
22+
23+
24+
def _monkey_patch():
25+
# NOTE(mdbooth): Anything imported here will not be monkey patched. It is
26+
# important to take care not to import anything here which requires monkey
27+
# patching.
28+
import eventlet
29+
import sys
30+
31+
# NOTE(mdbooth): Imports only sys (2019-01-30). Other modules imported at
32+
# runtime on execution of debugger.init().
33+
from nova import debugger
34+
35+
# Note any modules with known monkey-patching issues which have been
36+
# imported before monkey patching.
37+
# urllib3: https://bugs.launchpad.net/nova/+bug/1808951
38+
# oslo_context.context: https://bugs.launchpad.net/nova/+bug/1773102
39+
problems = (set(['urllib3', 'oslo_context.context']) &
40+
set(sys.modules.keys()))
41+
42+
# See https://bugs.launchpad.net/nova/+bug/1164822
43+
# TODO(mdbooth): This feature was deprecated and removed in eventlet at
44+
# some point but brought back in version 0.21.0, presumably because some
45+
# users still required it to work round issues. However, there have been a
46+
# number of greendns fixes in eventlet since then. Specifically, it looks
47+
# as though the originally reported IPv6 issue may have been fixed in
48+
# version 0.24.0. We should remove this when we can confirm that the
49+
# original issue is fixed.
50+
os.environ['EVENTLET_NO_GREENDNS'] = 'yes'
51+
52+
if debugger.enabled():
53+
# turn off thread patching to enable the remote debugger
54+
eventlet.monkey_patch(thread=False)
55+
elif os.name == 'nt':
56+
# for nova-compute running on Windows(Hyper-v)
57+
# pipes don't support non-blocking I/O
58+
eventlet.monkey_patch(os=False)
59+
else:
60+
eventlet.monkey_patch()
61+
62+
# NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet
63+
# hub use a monotonic clock to avoid issues with drifts of system time (see
64+
# LP 1510234 for details)
65+
# NOTE(mdbooth): This was fixed in eventlet 0.21.0. Remove when bumping
66+
# eventlet version.
67+
import oslo_service # noqa
68+
eventlet.hubs.use_hub("oslo_service:service_hub")
69+
70+
# NOTE(mdbooth): Log here instead of earlier to avoid loading oslo logging
71+
# before monkey patching.
72+
# NOTE(mdbooth): Ideally we would raise an exception here, as this is
73+
# likely to cause problems when executing nova code. However, some non-nova
74+
# tools load nova only to extract metadata and do not execute it. Two
75+
# examples are oslopolicy-policy-generator and sphinx, both of which can
76+
# fail if we assert here. It is not ideal that these utilities are monkey
77+
# patching at all, but we should not break them.
78+
# TODO(mdbooth): If there is any way to reliably determine if we are being
79+
# loaded in that kind of context without breaking existing callers, we
80+
# should do it and bypass monkey patching here entirely.
81+
if problems:
82+
from oslo_log import log as logging
83+
84+
LOG = logging.getLogger(__name__)
85+
LOG.warning("Modules with known eventlet monkey patching issues were "
86+
"imported prior to eventlet monkey patching: %s. This "
87+
"warning can usually be ignored if the caller is only "
88+
"importing and not executing nova code.",
89+
', '.join(problems))
90+
91+
# NOTE(mdbooth): This workaround is required to avoid breaking sphinx. See
92+
# separate comment in doc/source/conf.py. It may also be useful for other
93+
# non-nova utilities. Ideally the requirement for this workaround will be
94+
# removed as soon as possible, so do not rely on, or extend it.
95+
if (os.environ.get('OS_NOVA_DISABLE_EVENTLET_PATCHING', '').lower()
96+
not in ('1', 'true', 'yes')):
97+
_monkey_patch()

nova/test.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,7 @@
2121
2222
"""
2323

24-
import eventlet # noqa
25-
eventlet.monkey_patch()
24+
import nova.monkey_patch # noqa
2625

2726
import abc
2827
import copy

nova/tests/functional/__init__.py

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,4 @@
2020
:platform: Unix
2121
"""
2222

23-
import eventlet
24-
25-
eventlet.monkey_patch()
23+
import nova.monkey_patch # noqa

nova/utils.py

Lines changed: 0 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -49,10 +49,8 @@
4949
from oslo_utils import units
5050
import six
5151
from six.moves import range
52-
from six.moves import reload_module
5352

5453
import nova.conf
55-
from nova import debugger
5654
from nova import exception
5755
from nova.i18n import _, _LE, _LI, _LW
5856
import nova.network
@@ -1293,25 +1291,6 @@ def generate_hostid(host, project_id):
12931291
return ""
12941292

12951293

1296-
def monkey_patch():
1297-
if debugger.enabled():
1298-
# turn off thread patching to enable the remote debugger
1299-
eventlet.monkey_patch(thread=False)
1300-
elif os.name == 'nt':
1301-
# for nova-compute running on Windows(Hyper-v)
1302-
# pipes don't support non-blocking I/O
1303-
eventlet.monkey_patch(os=False)
1304-
else:
1305-
eventlet.monkey_patch()
1306-
1307-
# NOTE(rgerganov): oslo.context is storing a global thread-local variable
1308-
# which keeps the request context for the current thread. If oslo.context
1309-
# is imported before calling monkey_patch(), then this thread-local won't
1310-
# be green. To workaround this, reload the module after calling
1311-
# monkey_patch()
1312-
reload_module(importutils.import_module('oslo_context.context'))
1313-
1314-
13151294
if six.PY2:
13161295
nested_contexts = contextlib.nested
13171296
else:

0 commit comments

Comments
 (0)