Micronaut server is not working for clear text http2 (h2c)

[umutkocasarac]

I am trying to use h2c protocol on the Micronaut however my endpoint is not responding the request when I use http2. This is my configuration for the h2c

micronaut:
  application:
    name: test
  server:
    http-version: 2.0

Task List

• Steps to reproduce provided
• Stacktrace (if present) provided
• Example that reproduces the problem uploaded to Github
• Full description of the issue provided (see below)

Steps to Reproduce

Use the sample application from https://github.com/umutkocasarac/micronaut-h2c-test and run the application with ./gradlew run then make a curl request to the application

curl -v  --http2 http://localhost:8080/test
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /test HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
>
< HTTP/1.1 101 Switching Protocols
< connection: upgrade
< upgrade: h2c
* Received 101
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!

or this one

curl -v --http2-prior-knowledge  http://localhost:8080/test                                                                                                                                                         
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe61a80aa00)
> GET /test HTTP/2
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!

Expected Behaviour

It should upgrade the protocol to http2 and then respond the client

Actual Behaviour

Application freezes after the following log
Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!

Environment Information

• Micronaut Version: 2.3.3
• JDK Version: openjdk 11.0.8 2020-07-14

Example Application

https://github.com/umutkocasarac/micronaut-h2c-test

[umutkocasarac]

Please let me know if I should provide any further information for the issue.

[graemerocher]

Relates to #3389

Since browsers don't event support h2c currently Micronaut only supports h2 with HTTPS enabled. There are actually issues with the underlying Netty APIs that prevent us from fully supporting h2c

[umutkocasarac]

I see :(

Do you know the issue number on the Netty side, I have searched it but couldn't find it.
Thanks

[cbeams]

@graemerocher wrote:

Since browsers don't event support h2c currently Micronaut only supports h2 with HTTPS enabled.

True regarding browser support, but I assume many Micronaut users are building for machine-to-machine use cases where a browser is never involved. That's my use case anyway, and I'd like to get HTTP/2's compact wire format and connection reuse without being required to use TLS.

There are actually issues with the underlying Netty APIs that prevent us from fully supporting h2c

Could you elaborate? I researched this a bit today in the Netty repo, and I gather that folks are able to use Netty+h2c successfully elsewhere. Like the OP, I didn't see anything regarding "issues with [...] Netty APIs that prevent [...] fully supporting h2c"

For context, I'm vetting Micronaut for its suitability in implementing a certain system. h2c may end up being a hard requirement in that system, and I'd like to get a sense of how feasible it would be to implement that support ourselves if need be, i.e. to fix whatever needs fixing in Netty, etc. Thanks.

[graemerocher]

Hi Chris! Nice hearing from you, hope everything is well on your end. The issue is the same as this one with Spring spring-projects/spring-boot#19460 which points back to netty/netty#7079

Having said that it is a while since we looked this and maybe the underlying Netty APIs have evolved / been fixed so probably worth having another debugging session.

[graemerocher]

@yawkat next time you have cycles please take a crack at resolving this Netty issue

[cbeams]

Thanks, @graemerocher (and thanks in advance, @yawkat)!

Regarding the linked Spring and Netty issues, they talk about issues completing an h2c upgrade where a request has a body, e.g. a POST, and that such an issue on the Netty side can be solved through configuration.

In my case, though, I see the reported behavior when there is neither an upgrade request nor a request body involved.

Here are the relevant bits from my application.yml:

micronaut:
  application:
    name: repro
  server:
    port: 2040
    http-version: 2.0

For a baseline, let's observe that a vanilla HTTP/1.1 GET request works just fine:

$ curl -v http://localhost:2040/health
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 2040 (#0)
> GET /health HTTP/1.1
> Host: localhost:2040
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 Ok
< date: Fri, 1 Oct 2021 08:13:47 GMT
< Content-Type: application/json
< content-length: 15
< connection: keep-alive
<
* Connection #0 to host localhost left intact
{"status":"UP"}
* Closing connection 0

Now let's try the same GET request via HTTP/2, bypassing any upgrade negotiation with curl's --http2-prior-knowledge:

$ curl -v --http2-prior-knowledge http://localhost:2040/health
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 2040 (#0)
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x14600d600)
> GET /health HTTP/2
> Host: localhost:2040
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!

[... connection hangs indefinitely at this point ...]

[yawkat]

This particular bug should be fixed now, but there are still outstanding issues with the h2c support.

[cbeams]

Great, @yawkat, thanks! Looking forward to trying out the snapshot.

there are still outstanding issues with the h2c support

Could you elaborate? Is it reasonable to expect that many/most typical use cases should work under h2c now, or are the issues in question more significant than that?

[yawkat]

One issue I found is #6282. The h2c implementation (by necessity) has some pretty special properties because of the upgrade mechanism required, and this is leading to downstream issues like this. It is also not well-tested right now.

We discussed this internally and with potential changes on the standards and netty side of things h2c isn't really something to recommend. It is better to stick to normal http2.

Finally, the support for http2 in general is lacking on the feature side, e.g. the client connection pooling isn't ideal.

[cbeams]

I just tried out the fix, and the good news is that http2 over cleartext (h2c) now works as expected, e.g.:

$ curl -v --http2 http://localhost:2040/health
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 2040 (#0)
> GET /health HTTP/1.1
> Host: localhost:2040
> User-Agent: curl/7.64.1
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
>
< HTTP/1.1 101 Switching Protocols
< connection: upgrade
< upgrade: h2c
* Received 101
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 200
< date: Wed, 6 Oct 2021 09:59:19 GMT
< content-type: application/json
< content-length: 15
<
* Connection #0 to host localhost left intact
{"status":"UP"}* Closing connection 0

However, it works if and only if there is an upgrade request involved, as there was in the example above. If, alternatively, I make the equivalent request directly via HTTP/2 (see the --http2-prior-knowledge flag below), the response hangs as originally reported:

$ curl -v --http2-prior-knowledge http://localhost:2040/health
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 2040 (#0)
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x14400f800)
> GET /health HTTP/2
> Host: localhost:2040
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
[... hangs indefinitely ...]

[cbeams]

We discussed this internally and with potential changes on the standards and netty side of things h2c isn't really something to recommend. It is better to stick to normal http2.

Just to clarify, "normal http2" above is referring to h2, aka HTTP2-over-TLS, which is the only approach that browsers support today.

Given this constraint, it does indeed make sense in most cases to recommend that folks stay away from h2c and go with h2. I'd like to make the case, though, why it's still worth providing first-class support for h2c in Micronaut.

The first reason is for local network development use cases, where setting up self-signed certs, etc is just painful overhead.

The second reason is more specific to my application, though not necessarily unique to it. In my application, there are potentially many Micronaut instances in constant peer-to-peer communication with each other. This p2p communication happens via HTTP over Tor, where each Micronaut instance is a Tor onion service (aka Tor hidden service). As explained at https://security.stackexchange.com/a/75984, using TLS when communicating with Tor onion services is redundant, inefficient and harmful to privacy. So TLS is really not wanted here, but HTTP/2's tight wire format and long-lived connections very much are!

Particularly in a Tor environment, where establishing individual connections is extra expensive, having long-lived connections makes it feasible to pass lots of messages over HTTP. Without HTTP/2's connection reuse, one would probably have to drop down to building a custom protocol on top of raw sockets. Similarly, HTTP/2's header compression and binary wire format makes it feasible to pass around lots of HTTP messages efficiently. Without it, one would probably have to use e.g. protobufs or create a bespoke binary wire format. Being able to do all of these once custom, low-level things in plain-old-HTTP is hugely beneficial to developer productivity, system understandability, and implementation interoperability.

Again, I realize this use case is probably far from the norm for most Micronaut users, but I hope this context helps make it clear why using h2c in production can actually be a critical requirement. Thanks.

[yawkat]

I have created a separate issue for prior knowledge support.

I don't really see that the disadvantage of having to deal with self-signed certificates outweighs the security (even if it may seem superfluous in some cases) and simplicity advantage that TLS-based http2 has. I can see the privacy issues in your particular use case though. One solution would be to use a single certificate for all peers. There would still be some level of transport security because of TLS forward secrecy.

Either way, HTTP3 will not support an unencrypted mode from what I understand, and I don't know about the netty 5 support either. So I would avoid relying on h2c as a standard.