SSL socket errors

[hjwp]

We're seeing quite a few socket "Read error" and "Write error" in our logs. They're usually (but not absolutely always) associated with two C exceptions:

WARNING:Read error on 25: [Errno 1] _ssl.c:1331: error:140943FC:SSL routines:SSL3_READ_BYTES:sslv3 alert bad record mac

WARNING:Write error on 21: [Errno 1] _ssl.c:1217: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry

The latter looks similar to the error discussed in the comments in iostream.py, ie python bug 8240. Could it be a slightly different case of the same bug?
http://bugs.python.org/issue8240

When not associated with a C exception, we see only:

WARNING:Write error on 18: [Errno 14] Bad address

The first exception has a bit of google-juice too:
http://stackoverflow.com/questions/2603119/opening-ssl-urls-with-python
http://pythonpaste.org/news.html#id10

One thing that seems to help is running tornado from python 2.7 instead of python 2.6 - errors occur 5 times less frequently, in my last quick test (but sadly are not done away with completely)

Is there anything that can be done in the tornado code to try and avoid these? Or perhaps some way that we could use tornado differently to avoid them?

[bdarnell]

Hmm. Are you using ssl in HTTPServer or AsyncHTTPClient here? Do you know what's on the other end of the connection when you get these errors? The stackoverflow and paste links both mention threads; are you using threads in your application?

[hjwp]

Thanks for answering! We're using tornadio2, which uses WebSocketHandler from tornado... so I think that means neither, and no?

[bdarnell]

If you're using WebSocketHandler then the actual SSL support is still coming from HTTPServer and IOStream, although websockets are a more complicated case because they're full duplex (and browsers have tried doing some nonstandard things with them to mitigate the BEAST SSL attack). Since these are all C-level errors (the one you said was not a C exception was a socket.error instead of an ssl.error), there may not be much we can do about it at the tornado level. The "bad address" (EFAULT) error is actually the scariest one, since it suggests a stray pointer somewhere. I'm not sure how to debug this further.

[bdarnell]

I'm closing this bug because it is old and hasn't produced any actionable information; feel free to reopen if you're still having problems and/or can provide more details.

[bennettaur]

Can this be reopened? I'm running into this issue quite a bit. We use tornado on pypy and see the error on both platforms. I think pypy triggers it more because it has a tendency to move objects around in memory more.

Based on my digging, these all appear to be related:
http://stackoverflow.com/questions/9260937/unix-socket-error-14-efault-bad-address
http://bugs.python.org/issue8240
https://bitbucket.org/pypy/pypy/issues/1238/bad-write-retry-from-ssl-sockets

and it's documented in the tornado code itself:
https://github.com/tornadoweb/tornado/blob/master/tornado/iostream.py/#L839

However I don't think tornado handles the error appropriately here:
https://github.com/tornadoweb/tornado/blob/master/tornado/iostream.py/#L857

The error should be caught and the write_buffer_frozen, like it does for other cases and the write retried. I'll likely be experimenting with a fix soon, but reproducing can be somewhat random.

We're using SSLIOStreams when this is triggered

EDIT
I just realized bdarnell open the pypy issue regarding this.
Digging into it some more, this appears to be a bug in OpenSSL 1.0.2 on Mac OSX. This ruby thread appears to be related: excon/excon#467
They're solution was to simply warn about the issue and downgrade OpenSSL to 1.0.1

MORE EDITS
I can confirm I still see this issue in "OpenSSL 1.0.1f 6 Jan 2014" running with "PyPy 5.3.1 with GCC 4.8.2" using the PyPy Docker image pypy:2

As well as 'OpenSSL 1.0.2j 26 Sep 2016' on Mac OS X Version 10.11.6 (15G1004)