WebSockets support.

[jlaine]

Currently httpx is squarely focused on HTTP's traditional request / response paradigm, and there are well-established packages for WebSocket support such as websockets. In an HTTP/1.1-only world, this split of responsabilities makes perfect sense as HTTP requests / WebSockets work independently.

However, with HTTP/2 already widely deployed and HTTP/3 standardisation well under way I'm not sure the model holds up:

• implementations such as websockets are usually tied to HTTP/1.1 only, whereas httpx has support for HTTP/2 (and hopefully soon HTTP/3)
• we are missing out on the opportunity to multiplex "regular" HTTP request with WebSocket connections

Using the sans-IO wsproto combined with httpx's connection management we could provide WebSocket support spanning HTTP/1.1, HTTP/2 and HTTP/3. What are your thoughts on this?

One caveat: providing WebSocket support would only make sense using the AsyncClient interface.

[tomchristie]

So my assumption here would be "no", but that it's possible we'll want to expose just enough API in order to allow a websocket implementation to use httpx for handshaking the connection, and sharing the connection pool.

So, to the extent that we might allow it to be possible, I'd expect that to be in the form of a third party package, that has httpx as a dependency.

I'm not sure exactly what API that'd imply that we'd need to support, but it's feasible that we might end up wanting to expose some low-level connection API specifically for supporting this kind of use case. The most sensible tack onto thrashing out what we'd need there would be a proof-of-concept implementation that'd help us concretely indentify how much API we'd need to make that possible.

Does all that make sense, or am I off on the wrong track here?

[jlaine]

What you are saying does make sense, I haven't quite made up my own mind on this.

Pros of a third-party package handling the websockets:

• keeps httpx's focus on request / response
• allows for competing implementations of the WebSocket part

Cons:

• we need to make some fairly complex APIs public, which might limit our ability to refactor httpx's internals
• we are going to have to expose structures which are very specific to the HTTP version
• we somewhat break the symmetry with httpx's server counterparts which do a good job of handling HTTP + WebSockets behind a single API

[tomchristie]

Interesting points, yeah. I guess I'd be open to reassessing this as we go.

We'd need to identify what the API would look like, both at the top user-facing level, and at whatever new lower-level cutoffs we'd need to be introducing.

[sethmlarson]

I'm in favor of adding websockets definitely, would be a good feature to work towards in a v2? Still so much to do to get a v1 released.

Thanks for bringing this up @jlaine! I've not had to deal with websockets so I didn't even think they'd be at home in a library like HTTPX. :)

[tomchristie]

I'm in favor of adding websockets definitely, would be a good feature to work towards in a v2? Still so much to do to get a v1 released.

Probably not a bad call, yeah.

[jlaine]

I'm in favor of adding websockets definitely, would be a good feature to work towards in a v2? Still so much to do to get a v1 released.

I'll see if I can find time to put together a minimal PR supporting HTTP/1.1 and HTTP/2 to scope out what this entails. Obviously if there are more pressing needs for v1 there is zero pressure to merge it.

@aaugustin just pinging you so you're in the loop : this is not a hostile move against websockets, and your insights would be most welcome

[aaugustin]

I'm currently working on refactoring websockets to provide a sans-I/O layer. If it's ready within a reasonable timeframe, perhaps you could use it. It will handle correctly a few things that wsproto doesn't, if the comments I'm seeing in wsproto's source code are correct.

One of my issues with sans-I/O is the prohibition async / await, making it impossible to provide a high-level API. For this reason, I'm not sure that sans-I/O as currently specified is the best approach. This makes me hesitate on a bunch of API design questions...

Building an API that httpx will consume (either strict sans-I/O or something else) would be a great step forwards. Perhaps we can cooperate on this?

---

Also I would really like websockets to work over HTTP/2 and HTTP/3.

I'm quite happy with my implementation of a subset of HTTP/1.1 — I'm not kidding, even though it's home-grown, I honestly think it's all right. It may be possible to achieve HTTP/2 in the same way but it will be partial e.g. it will be impossible to multiplex websockets and HTTP on the same connection. Given what I've seen of aioquic, I don't think it's reasonable to do HTTP/3 that way.

So I'm interested in working with third-party packages that handle HTTP/2 and HTTP/3 to figure out what that entails.

[ClericPy]

As for me, httpx is not only the next generation HTTP client for requests but also aiohttp.

So expecting a new choice to take the place of aiohttp's websocket. https://github.com/ftobia/aiohttp-websockets-example/blob/master/client.py

Best wishes to the author, this lib is a great help for me.

PS: aiohttp is good enough, except some frequently changing api protocol (like variable names), which raised a backward incompatibility error by a new version.

[aaugustin]

FYI I started implementing a Sans-I/O layer for websockets: python-websockets/websockets#466

It may not be obvious from the first commits because I have to untangle a lot of stuff from asyncio before I can get anything done :-)

[sethmlarson]

@aaugustin That's awesome news! :) Thanks for updating this thread. It'd be interesting to see how an upgrade from h11 into the websockets state-machine might look. Something to definitely write up a POC for that little exchange.

(Also, congrats on the Tidelift sponsorship!)

[aaugustin]

So, here's how I think the bridge could work.

Since we're talking about httpx, I'm focusing on the client side here, but the same logic would apply on the server side.

In addition to the regular APIs that receive and send bytes, websockets should provide an API to receive and send already parsed HTTP headers. This API will support bolting websockets on top of any HTTP/1.1 implementation.

In addition to:

from websockets import ClientConnection

ws = ClientConnection()
bytes_to_send = ws.connect()
...
ws.receive_data(bytes_received)

I need to provide something like:

from websockets import ClientConnection

ws = ClientConnection()
request = ws.build_handshake_request()
# here request is (path, headers) tuple
bytes_to_send = serialize(request)  # <-- you can do this with any library you want
...
response = parse(bytes_received)  # <-- you can do this with any library you want
# here response is a (status_code, reason_phrase, headers) tuple
ws.receive_handshake_response()

The latter happens under the hood anyway so it's clearly doable.

It's "just" a matter of naming things :-) — which I care deeply about.

---

WebSocket over HTTP/2 requires a different handshake, so websockets will need another APIs to support it. I have no idea about WebSocket over HTTP/3.

[jlaine]

WebSockets over HTTP/3 haven't been officially specified, but it is extremely likely it will work like for HTTP/2 (RFC 8441), namely using a :protocol pseudo-header. This is what I have assumed for the aioquic demo client + server, and I believe @pgjones did the same for hypercorn.

[tomchristie]

I'd like us to treat this as out-of-scope at this point in time.

Yes I think we'll want to provide enough API to make this do-able at some point in the not too distant future, but being able to do that while still aiming for a sqaured away API stable 1.0 release isn't something we're able to do just yet.