New SSL implementation based on ssl.MemoryBIO

[pitrou]

BPO	22560
Nosy	@gvanrossum, @pitrou, @vstinner, @giampaolo, @1st1
Files	sslproto.patch sslproto3.patch sslproto-4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-01-15.08:46:17.477>
created_at = <Date 2014-10-05.22:28:22.765>
labels = ['type-feature', 'expert-asyncio']
title = 'New SSL implementation based on ssl.MemoryBIO'
updated_at = <Date 2015-03-23.15:00:17.902>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2015-03-23.15:00:17.902>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2015-01-15.08:46:17.477>
closer = 'vstinner'
components = ['asyncio']
creation = <Date 2014-10-05.22:28:22.765>
creator = 'pitrou'
dependencies = []
files = ['36821', '36945', '37638']
hgrepos = []
issue_num = 22560
keywords = ['patch']
message_count = 28.0
messages = ['228628', '228632', '228634', '228652', '229507', '229835', '229916', '232169', '233282', '233283', '233299', '233300', '233301', '233302', '233303', '233306', '233311', '233617', '233618', '233631', '233697', '233971', '233972', '233973', '234041', '234064', '234065', '234066']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'geertj', 'pitrou', 'vstinner', 'giampaolo.rodola', 'python-dev', 'sbt', 'yselivanov']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22560'
versions = ['Python 3.5']

[pitrou]

Now that bpo-21965 is implemented, it is possible to improve SSL support in asyncio by making it independent of how the underlying event loop works (e.g. whether it is a Unix-like reactor or a proactor).

[pitrou]

Here is a proof-of-concept patch. I've only tested it under Linux, but it should be possible to write a simple _make_ssl_transport() for the BaseProactorEventLoop.

[gvanrossum]

This is awesome news!

Since this is 3.5 only, I guess this means the end of my attempts to keep the asyncio source code identical in the Tulip repo (from which I occasionally create builds that work with Python 3.3) and in the 3.4 and 3.5 branches. I guess that's okay, people should be switching to 3.5 anyway. I'm not sure what approach to follow to keep the branches at least somewhat in sync -- perhaps tulip and 3.4 can still be kept identical, with changes merged from 3.4 into 3.5 (default) but 3.5 differing in some places? Or perhaps the code can be kept identical with the exception of the sslproto.py file, and conditional import of the latter?

[pitrou]

Or perhaps the code can be kept identical with the exception of the
sslproto.py file, and conditional import of the latter?

I think that's reasonable, yes. The _SelectorSslTransport is still there and can be used if the ssl module is not recent enough.

[pitrou]

Here is an updated patch. It hooks into the Proactor event loop (tested under Windows) and also adds a fallback for older Pythons (with tests).

[pitrou]

Does someone want to review this?

[vstinner]

I will try to take a look next week.

[pitrou]

From bpo-22768:

"""

Maybe
transport.get_extra_info('socket').getpeercert(True)
would be okay, no patch needed?

That will be problematic with bpo-22560. The clear-text socket object and the SSL object become unrelated, and it would be logical for get_extra_info('socket') to return the clear-text socket, so either a get_extra_info('ssl') would be needed, or we should expose the SSL properties directly as extra info members.
"""

[pitrou]

Ping :-)

[pitrou]

Note this could probably help https://twitter.com/icgood/status/549915951165358080, which Victor seems to care about :-)

[gvanrossum]

Maybe we should just accept this without review? I really don't have time to review 600+ lines of code, sorry.

[vstinner]

Maybe we should just accept this without review? I really don't have time to review 600+ lines of code, sorry.

SSL/TLS is very important and the patch is large, a review is required. I posted a first review with a lot of comments.

[vstinner]

Sorry for the delay. I understood that the change targets the proactor event loop, and I was busy to fix annoying random bugs in this code (it's not done yet, see for example the issue bpo-23095 for the most recent bug). Windows is not my favorite OS, I am less intersted to fix Windows specific bugs, but sometimes I try to fix them.

I still don't understand if the change is (or should be) specific to the proactor event loop or not. Antoine, can you please elaborate the rationale of your patch?

Is the "legacy" code only used on Python 3.4 and older? Is ssl.MemoryBIO always present in Python 3.5 and newer?

I would like to see benchmarks of memory BIO vs current code on Linux and Windows. Even if it would make the code simpler to always use memory BIO, I care of network performances. Maybe we may only use memory BIO for the proactor event loop?

Do you know if other applications use memory BIO for networking with SSL?

Did you try your patch on Python 3.3? On Linux and Windows?

Anyway, great job!