asyncio backend

[Lukasa]

It would be awesome if we could have an asyncio-based backend that exposes an optional synchronous API. I have no idea if we can do this or not but I'd love it if we could, it would clean the code up tremendously.

[piotr-dobrogost]

That's definitely the way to go.

[sigmavirus24]

asyncio will be rough. As I understand it (from a recent post to python-dev) the docs are barely there. This would be an awesome feature though.

[Lukasa]

Yeah, I acknowledge that the lack of docs are a problem. I'm not going to really attack this until 3.4 happens, but I'm pretty hopeful that I can make this work.

[Lukasa]

Based on this article here, all we need to do is to have our synchronous wrapper APIs call this:

loop = asyncio.get_event_loop()
loop.run_until_complete(print_page('http://example.com'))

[Lukasa]

Having concluded that we can definitely do this, it's time to do some API design. What's a good asynchronous API going to look like?

First, let's all acknowledge that the request()/getresponse() API is totally stupid, and we're only doing it for backward compatibility. When using asyncio directly, you want to just be able to make a single call. So we map that onto a new method, called request(), which just straight-up returns a response. This will be a coroutine, which means you interact with it like this:

>>> conn = HTTP20Connection('twitter.com:443')
>>> resp = yield from conn.request('POST', '/some/path', headers=headers, body=body)

The incremental 'build it as you go' API is one that I'm open to keeping, especially as it's very useful for sending data in chunks, but it's not really well designed for HTTP/2.0 at the moment. My best idea ATM is to refactor it into three methods: send_headers(), send_data() and get_response() (ah crap, it's back). This at least requires that you work out what all your headers are beforehand. Interestingly, all of these will need to be coroutines because we want to make sure that failures get thrown into the correct context. This will allow the following interface:

>>> conn = HTTP20Connection('twitter:443')
>>> stream_id = yield from conn.send_headers(headers)
>>> for chunk in some_slow_generator():
>>>    yield from conn.send_data(chunk)
>>> resp = yield from conn.get_response()

It'd be useful to get input from people who might actually end up interacting with this interface: how'd you like it to look? /cc @sigmavirus24, @shazow, @piotr-dobrogost. /cc @kennethreitz if he finds time, as his input would be hugely valuable.

[Lukasa]

Hmm, I've got a large quantity of the asyncio code worked out but I have a busy-wait loop. That's a bit sad.

[Lukasa]

Ok, I've fixed that up (hopefully) and gotten to the stage where I can start running some tests. This doesn't actually work because the tests all use the old synchronous API and I haven't implemented a wrapper for it yet. That's the next work item.

[Lukasa]

It's possible that there are interesting developments in Python that might affect my approach to this. Watch this space.

[sigmavirus24]

watching intensifies

[Lukasa]

So, PEP 466 is what I was referring to. If this gets accepted then it opens the door for Python 2.7 to have an ssl module that allows for an actual HTTP/2 stack: I think I'd rather have that than asyncio support.

[Lukasa]

We've added support for Python 2.7, which makes this a no-go.