Using both eventlet.monkey_patch() and eventlet.wsgi.server() together with SSL fails with ssl.SSLWantReadError

[acaos]

The following simple test script fails (when connected to using the openssl command-line client, or from a browser):

#!/usr/bin/env python3
#
# To test breakage:
#   $ openssl s_client -crlf -connect 127.0.0.1:8448
#

import eventlet
eventlet.monkey_patch()

import sys

from eventlet import wsgi

def hello_world (env, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    return ['Hello, World!\r\n']


def main ():
    listener = eventlet.wrap_ssl(
        eventlet.listen(('127.0.0.1', 8448)),
        certfile='test.crt',
        keyfile='test.key',
        server_side=True)

    wsgi.server(listener, hello_world)

if __name__ == '__main__':
    sys.exit(main())

with the following exception chain:

Traceback (most recent call last):
  File "foo.py", line 31, in <module>
    sys.exit(main())
  File "foo.py", line 28, in main
    wsgi.server(listener, hello_world)
  File "/usr/lib/python3.5/site-packages/eventlet/wsgi.py", line 864, in server
    client_socket = sock.accept()
  File "/usr/lib/python3.5/site-packages/eventlet/green/ssl.py", line 335, in accept
    suppress_ragged_eofs=self.suppress_ragged_eofs)
  File "/usr/lib/python3.5/site-packages/eventlet/green/ssl.py", line 90, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/site-packages/eventlet/green/ssl.py", line 243, in do_handshake
    super(GreenSSLSocket, self).do_handshake)
  File "/usr/lib/python3.5/site-packages/eventlet/green/ssl.py", line 108, in _call_trampolining
    return func(*a, **kw)
  File "/usr/lib/python3.5/ssl.py", line 983, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 628, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:646)

I've tracked this down to the set_nonblocking(newsock) call in the GreenSSLSocket.accept() method in eventlet/green/ssl.py around line 318. The issue is fixed by either commenting out this call or by not calling eventlet.monkey_patch() at the start of the script (the second not really being an option for our use case).

Of course, I'm not sure what the adverse effects of removing the set_nonblocking() call might be, or if there is some other, better solution to the problem.

$ python3 -V
Python 3.5.0
$ python3 -c 'import eventlet; print(eventlet.__version__)'
0.18.4

[temoto]

Ouch, that's a very unhappy situation.
Please, try eventlet master, it has relevant code modified, could fix this problem.

Also, my deepest recommendation: do not use Python for SSL/TLS public servers. Put a reliable TLS terminator software (like haproxy, nginx, stunnel, stud) in front to sleep well.

[acaos]

Just wanted to confirm that this still happens using eventlet 0.19.0 (with the fix).

(and this is being used for a private server with no access to the Internet at large - your warning is well taken and quite correct).

[thomasgoirand]

Hi. I'm the maintainer of all OpenStack packages in Debian. Using OpenStack Glance, and eventlet 0.20.0 or 0.23.0 (I tried both versions), I have the same problem. It works ok without SSL, and it seems to also work with Python 2.7 (though I couldn't test this).

[temoto]

Thanks for ping, I will look at this today.

[temoto]

wsgi excluded version.

setblocking is definitely one to blame, but probably need a bit smarter move than removing it.

#!/usr/bin/env python3
import eventlet
eventlet.monkey_patch()

def srv(l):
    s, _ = l.accept()
    s.settimeout(5)
    r = s.makefile('rb')
    w = s.makefile('wb')
    r.readline(1<<10)
    r.readline(2<<10)
    w.write(b'HTTP/1.0 200 OK\r\n\r\n')
    w.flush()


def cli(sn):
    s = eventlet.connect(sn)
    s = eventlet.wrap_ssl(s, server_side=False)
    r = s.makefile('rb')
    w = s.makefile('wb')
    w.write(b'GET / HTTP/1.0\r\n\r\n')
    w.flush()
    status = r.readline(1<<10)
    assert status == b'HTTP/1.0 200 OK\r\n', status


def main ():
    listener = eventlet.wrap_ssl(
        eventlet.listen(('127.0.0.1', 8448)),
        certfile='tests/test_server.crt',
        keyfile='tests/test_server.key',
        server_side=True,
    )
    t2 = eventlet.spawn(cli, listener.getsockname())
    srv(listener)
    t2.wait()
    print('pass')

if __name__ == '__main__':
    main()

[temoto]

Workaround version. Warning: SSL IO may block everything. Better tested version is coming tonight.

pip install https://github.com/eventlet/eventlet/archive/339c7ad4eaa4766ac8135fd13eed3adeaa43b276.zip

[zhur0ng]

@temoto we aslo faceing this issue using eventlet 0.25.1, do we have any plan about this issue?

[temoto]

@zhur0ng have you tried 339c7ad ?

[zhur0ng]

@temoto thanks for your quickly reply, I have tried 339c7ad, api service can works, but still have errors print in logs.
8]: read = self.read(nbytes, buffer_)
8]: File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 176, in read
8]: super(GreenSSLSocket, self).read, *args, **kwargs)
8]: File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
8]: return func(*a, **kw)
8]: File "/usr/lib64/python3.6/ssl.py", line 830, in read
8]: return self._sslobj.read(len, buffer)
8]: File "/usr/lib64/python3.6/ssl.py", line 587, in read
8]: v = self._sslobj.read(len, buffer)
8]: OSError: [Errno 0] Error

[temoto]

@zhur0ng and same code without eventlet.monkey_patch() works without errors?

[zhur0ng]

@temoto sorry, Our service using monkey_patch, so I just care about the code with monkey_patch.

[temoto]

@zhur0ng I'm trying to narrow down the problem. Can you help?

[zhur0ng]

@temoto sure, I can help this.

[temoto]

@zhur0ng so please tell me if you see that OSError: [Errno 0] Error with 339c7ad but without monkey_patch.

[zhur0ng]

@temoto I remove the monkey_patch code, still have this error.

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
timer()
File "/usr/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in call
cb(*args, **kw)
File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
result = function(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 818, in process_request
proto.init(conn_state, self)
File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 357, in init
self.handle()
File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 390, in handle
self.handle_one_request()
File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 419, in handle_one_request
self.raw_requestline = self._read_request_line()

File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 402, in _read_request_line
return self.rfile.readline(self.server.url_length_limit)

File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 241, in recv_into
return self.base_recv(nbytes, flags, into=True, buffer=buffer)
File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 256, in base_recv
read = self.read(nbytes, buffer)
File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 176, in read
super(GreenSSLSocket, self).read, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/lib64/python3.6/ssl.py", line 830, in read
return self._sslobj.read(len, buffer)

File "/usr/lib64/python3.6/ssl.py", line 587, in read
v = self._sslobj.read(len, buffer)
OSError: [Errno 0] Error

[temoto]

Okay that didn't help because eventlet ssl was still used, but I found where errno 0 come from and half confident it is not related to this issue so can publish the fix.

@zhur0ng you may want to subscribe to this https://bugs.python.org/issue31122

[temoto]

Fix was merged into master 4501932 please reopen if SSLWantReadError still present.

[lingxiankong]

@temoto Thanks for the fix, that works for me. I'm wondering when are you available for doing a release?

[temoto]

@lingxiankong 0.25.2 uploaded

[lingxiankong]

@lingxiankong 0.25.2 uploaded

Thanks so much!