New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_SSLErrorHandshakeTimeout exception crashed whole web server #1295

Closed
BobDenar1212 opened this Issue Oct 18, 2018 · 1 comment

Comments

Projects
None yet
2 participants
@BobDenar1212

BobDenar1212 commented Oct 18, 2018

  • gevent version: 1.3.6
  • Python version: Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)] on win32
  • Operating System: Win10 version 1709 (OS Build 16299.611)
  • Flask version: 1.0.2

Description:

Hi,

A Flask application is served by gevent WSGIServer with ssl certificate trusted by an inhouse CA.
Everything works perfectly until the following chained exceptions occur and stop the web application. I have no idea what is causing this error since there is no obvious pattern.

Sorry if this not the place to submit this, but I could'nt find any relevant case after a long googling.

2018-10-18 11:57:16.070000 CRITICAL flask.app Traceback (most recent call last):
  File "src\gevent\greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "C:\Program Files\Python36\lib\site-packages\gevent\baseserver.py", line 26, in _handle_and_close_when_done
    return handle(*args_tuple)
  File "C:\Program Files\Python36\lib\site-packages\gevent\server.py", line 201, in wrap_socket_and_handle
    with _closing_socket(self.wrap_socket(client_socket, **self.ssl_args)) as ssl_socket:
  File "C:\Program Files\Python36\lib\site-packages\gevent\_ssl3.py", line 693, in wrap_socket
    ciphers=ciphers)
  File "C:\Program Files\Python36\lib\site-packages\gevent\_ssl3.py", line 267, in __init__
    raise x
  File "C:\Program Files\Python36\lib\site-packages\gevent\_ssl3.py", line 263, in __init__
    self.do_handshake()
  File "C:\Program Files\Python36\lib\site-packages\gevent\_ssl3.py", line 592, in do_handshake
    self._wait(self._read_event, timeout_exc=_SSLErrorHandshakeTimeout)
  File "src\gevent\_hub_primitives.py", line 265, in gevent.__hub_primitives.wait_on_socket
  File "src\gevent\_hub_primitives.py", line 266, in gevent.__hub_primitives.wait_on_socket
  File "src\gevent\_hub_primitives.py", line 261, in gevent.__hub_primitives._primitive_wait
  File "src\gevent\_hub_primitives.py", line 262, in gevent.__hub_primitives._primitive_wait
  File "src\gevent\_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src\gevent\_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src\gevent\_hub_primitives.py", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src\gevent\_waiter.py", line 151, in gevent.__waiter.Waiter.get
  File "src\gevent\_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src\gevent\_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src\gevent\_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src\gevent\__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
socket.timeout: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\site-packages\gevent\_ffi\loop.py", line 126, in python_callback
    the_watcher.callback(*args)
  File "src\gevent\greenlet.py", line 504, in gevent._greenlet.Greenlet.throw
  File "src\gevent\greenlet.py", line 718, in gevent._greenlet.Greenlet.run
  File "src\gevent\greenlet.py", line 706, in gevent._greenlet.Greenlet._report_error
  File "C:\Program Files\Python36\lib\site-packages\gevent\hub.py", line 474, in handle_error
    self.print_exception(context, type, value, tb)
  File "C:\Program Files\Python36\lib\site-packages\gevent\hub.py", line 546, in print_exception
    errstream.write('%s failed with %s\n\n' % (context, getattr(type, '__name__', 'exception'), ))
AttributeError: 'NoneType' object has no attribute 'write'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\site-packages\gevent\baseserver.py", line 364, in serve_forever
    self._stop_event.wait()
  File "src\gevent\event.py", line 240, in gevent._event.Event.wait
  File "src\gevent\event.py", line 140, in gevent._event._AbstractLinkable._wait
  File "src\gevent\event.py", line 117, in gevent._event._AbstractLinkable._wait_core
  File "src\gevent\event.py", line 119, in gevent._event._AbstractLinkable._wait_core
  File "src\gevent\_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src\gevent\_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src\gevent\_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src\gevent\__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
  File "C:\Program Files\Python36\lib\site-packages\gevent\hub.py", line 565, in run
    loop.run()
  File "C:\Program Files\Python36\lib\site-packages\gevent\libuv\loop.py", line 477, in run
    ran_callbacks = self.__run_queued_callbacks()
  File "C:\Program Files\Python36\lib\site-packages\gevent\libuv\loop.py", line 442, in __run_queued_callbacks
    val = _callbacks.python_callback(handle, arg)
  File "C:\Program Files\Python36\lib\site-packages\gevent\_ffi\loop.py", line 140, in python_callback
    the_watcher.loop._keepaliveset.add(the_watcher)
AttributeError: 'NoneType' object has no attribute '_keepaliveset'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python36\Scripts\al-khwarizmi.py", line 50, in main
    self._http_server.serve_forever()
  File "C:\Program Files\Python36\lib\site-packages\gevent\baseserver.py", line 366, in serve_forever
    Greenlet.spawn(self.stop, timeout=stop_timeout).join()
  File "src\gevent\greenlet.py", line 661, in gevent._greenlet.Greenlet.join
  File "src\gevent\greenlet.py", line 687, in gevent._greenlet.Greenlet.join
  File "src\gevent\greenlet.py", line 678, in gevent._greenlet.Greenlet.join
gevent.exceptions.InvalidSwitchError: Invalid switch into Greenlet.join(): ()

2018-10-18 11:57:16.054000 CRITICAL flask.app Invalid switch into Greenlet.join(): ()

Any help would be much appreciated

@jamadden

This comment has been minimized.

Member

jamadden commented Oct 31, 2018

This appears to be happening during interpreter shutdown. (OR your application has heavily customized gevent's hub.)

  File "C:\Program Files\Python36\lib\site-packages\gevent\hub.py", line 474, in handle_error
    self.print_exception(context, type, value, tb)

We get here to handle an exception deemed to be not an application error. By default that's SystemExit (or GreenletExit). (An application could change that.)

    errstream.write('%s failed with %s\n\n' % (context, getattr(type, '__name__', 'exception'), ))
AttributeError: 'NoneType' object has no attribute 'write'

errstream comes from hub.exception_stream which is a property that returns sys.stderr. During shutdown, values in sys are set to None (An application could replace exception_stream to do something different.)

    the_watcher.loop._keepaliveset.add(the_watcher)
AttributeError: 'NoneType' object has no attribute '_keepaliveset'

This means that the_watcher has already been stopped and closed.

  File "src\gevent\greenlet.py", line 678, in gevent._greenlet.Greenlet.join
gevent.exceptions.InvalidSwitchError: Invalid switch into Greenlet.join(): ()

The hub died badly due to previous errors. At the top level, I would tend to expect the process to exit at that point. If that's not the top level, well, with the hub dead, nothing much is going to happen.

It's possible that a timeout error (any error happening in a non-hub greenlet), happening at the same time as interpreter shutdown, could lead to such a situation. I would expect such a situation to be rare and difficult to reproduce, though.

If that's the case, in this particular scenario there are things we can do to try to avoid using interpreter state during shutdown, but that's not always easy.

jamadden added a commit that referenced this issue Oct 31, 2018

jamadden added a commit that referenced this issue Oct 31, 2018

jamadden added a commit that referenced this issue Oct 31, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment