Feature/http2 prior knowledge

[p1-alexandrevaney]

Feature

This PR aims to enforce the use of HTTP/2 by the AsyncHTTPClient without the need for HTTP/1 Upgrade.

Currently 5G Core network functions communicate in h2c, this httpcore feature aims to be extended in httpx to help communicating with 5GC Network Functions without TLS.

Description

When the self.http1 attribute is set to False along with the self.http2 attribute set to True, the connection instantiated is forced to AsyncHTTP2Connection

I also added an exception in case the http2 stack receives an HTTP/1 Bad Request which is the answer you get when sending http2 request to a server that does not support it.

For https requests, the alpn protocol http/1.1 is removed from the ssl_context object if the self.http1 attribute is set to False along with the self.http2 attribute set to True

[tomchristie]

Okay, so as mentioned on encode/httpx#1523 I think an http1=[True|False], http2=[True|False] interface probably makes more sense for us. The other thing is that I don't think this change should really dig all the way down into the networking backends. It looks to me like it could instead just take effect at the level of the SyncHTTPConnection()/AsyncHTTPConnection() and effect the logic of the _create_connection() method.

[p1-alexandrevaney]

I have replaced the http2_prior_knowledge attribute with an http1 one and used the simpler approach that you mentioned.

I can maybe throw an error if both http1 and http2 are set to False on initialization of the (Async/SyncHTTP)Connection.

[tomchristie]

Okay, so the thing that'd be really helpful here would be a concrete real-world example of a URL that you'd want to use this against. That'll help us verify what we're doing with an actually use case, rather than "in theory".

If you can walk us through exactly what it is you're trying to do in practice, it's a huge help in adequately describing the motivation for any code changes, and helps ground us in real use-cases rather than just indulging in bit wrangling for the sake of it.

I started doing a bit more digging into this, and I can see that it's actually a bit more involved that simply "forcibly use HTTP/2 regardless of any ALPN negotiation."...

So... https://tools.ietf.org/html/rfc7540#section-3.4

3.4. Starting HTTP/2 with Prior Knowledge

A client can learn that a particular server supports HTTP/2 by other means. For example, [ALT-SVC] describes a mechanism for advertising this capability.

A client MUST send the connection preface (Section 3.5) and then MAY immediately send HTTP/2 frames to such a server; servers can identify these connections by the presence of the connection preface. This only affects the establishment of HTTP/2 connections over cleartext TCP; implementations that support HTTP/2 over TLS MUST use protocol negotiation in TLS [TLS-ALPN].

Likewise, the server MUST send a connection preface (Section 3.5).

Without additional information, prior support for HTTP/2 is not a strong signal that a given server will support HTTP/2 for future connections. For example, it is possible for server configurations to change, for configurations to differ between instances in clustered servers, or for network conditions to change.

And this... https://tools.ietf.org/html/rfc7540#section-3.5

3.5. HTTP/2 Connection Preface

In HTTP/2, each endpoint is required to send a connection preface as a final confirmation of the protocol in use and to establish the initial settings for the HTTP/2 connection. The client and server each send a different connection preface.

The client connection preface starts with a sequence of 24 octets, which in hex notation is:

0x505249202a20485454502f322e300d0a0d0a534d0d0a0d0a

That is, the connection preface starts with the string "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"). This sequence MUST be followed by a SETTINGS frame (Section 6.5), which MAY be empty. The client sends the client connection preface immediately upon receipt of a 101 (Switching Protocols) response (indicating a successful upgrade) or as the first application data octets of a TLS connection. If starting an HTTP/2 connection with prior knowledge of server support for the protocol, the client connection preface is sent upon connection establishment.

Note: The client connection preface is selected so that a large proportion of HTTP/1.1 or HTTP/1.0 servers and intermediaries do not attempt to process further frames. Note that this does not address the concerns raised in [TALKING].

The server connection preface consists of a potentially empty SETTINGS frame (Section 6.5) that MUST be the first frame the server sends in the HTTP/2 connection.

The SETTINGS frames received from a peer as part of the connection preface MUST be acknowledged (see Section 6.5.3) after sending the connection preface.

To avoid unnecessary latency, clients are permitted to send additional frames to the server immediately after sending the client connection preface, without waiting to receive the server connection preface. It is important to note, however, that the server connection preface SETTINGS frame might include parameters that necessarily alter how a client is expected to communicate with the server. Upon receiving the SETTINGS frame, the client is expected to honor any parameters established. In some configurations, it is possible for the server to transmit SETTINGS before the client sends additional frames, providing an opportunity to avoid this issue.

Clients and servers MUST treat an invalid connection preface as a connection error (Section 5.4.1) of type PROTOCOL_ERROR. A GOAWAY frame (Section 6.8) MAY be omitted in this case, since an invalid preface indicates that the peer is not using HTTP/2.

[p1-alexandrevaney]

Sure, what i am trying to do is send messages to several 5g core network equipments that i simulated with open source implementations such as https://github.com/open5gs/open5gs.

In 5G, core network equipments are exchanging with http2 messages. They do not support http1 and most do not use TLS (some equipments are gonna use TLS in inter-network exchanges, for example in roaming scenarios).

When i tried to use the http2 setting without TLS, httpcore/httpx did not take it into account since it only uses the http2 upgrade mechanism (which cannot be used in my case since 5g equipments do not answer to http1 messages)

For the connection preface talked about in the RFC, it seems httpcore already uses it:
httpcore call to h2 initiate_connection
h2 initiate_connection

For the TLS case, it seems that when trying to negociate alpn h2 with a server that does not support it, the server will just answer in http1, triggering httpcore.RemoteProtocolError: HTTP/1.1 received

This is an example of a message that went through with curl

curl --http2-prior-knowledge -X GET "http://127.0.0.1:7777/nnrf-nfm/v1/nf-instances?nf-type=AMF&limit=10" -H  "accept: application/3gppHal+json"

[tomchristie]

Okay, I've had a bit of a look through their docs, but there's quite a lot to get through.

What's the best resource for getting up to the point that I'd be able to run this...

curl --http2-prior-knowledge -X GET "http://127.0.0.1:7777/nnrf-nfm/v1/nf-instances?nf-type=AMF&limit=10" -H  "accept: application/3gppHal+json"

...against the simulated network?

[p1-alexandrevaney]

Quickest way would be to follow https://open5gs.org/open5gs/docs/guide/01-quickstart/ section 2 : Install Open5GS with a Package Manager. It is quite fast on ubuntu:

$ sudo apt update
$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:open5gs/latest
$ sudo apt update
$ sudo apt install open5gs

then sudo systemctl restart open5gs-* to start all virtualized equipments

and then you can curl the 5GC NRF equipment which is listening on 127.0.0.10:7777 with

curl --http2-prior-knowledge -X GET "http://127.0.0.10:7777/nnrf-nfm/v1/nf-instances?nf-type=AMF&limit=10" -H "accept: application/3gppHal+json"

[tridentsx]

What is the current status of getting http2 working in httpx ?
We have been using the pull requests for a few weeks without issues ? Will they be rebased and merged with httpx or with there be another solution for http2 prior knowledge ?

[bli74]

We use this pull request also together with a small change in httpx to forward parameter 'http1' to httpcore.
These changes are used as a base for a Robotframework library with HTTP/2 support.
So far everything is working fine.

[tomchristie]

This check is way too broad. For example, it could end up raising an error if the response body happened to include literal the string "HTTP/1.1" as part of the actual document content.

I think we want to remove these.

[p1-alexandrevaney]

Fixed in this commit

[tomchristie]

Does anybody have a live URL that we could use to test this out against?

Alternatively what's the best way for us to test this? Using hypercorn perhaps?

[tomchristie]

Looks like it... https://gitlab.com/pgjones/hypercorn/-/issues/160

[tomchristie]

Great stuff, @p1-alexandrevaney! I've followed up on this with some minor tweaks: #333

[tridentsx]

Does anybody have a live URL that we could use to test this out against?

Alternatively what's the best way for us to test this? Using hypercorn perhaps?

http://http2.pro

I think this works