Http2Exception frame length exceeds maximum

[mfirry]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.6.1

What did you expect to see?

We noticed a sudden disconnection and the only log I could grab from is the following coming from the client side

Caused by: io.netty.handler.codec.http2.Http2Exception: Frame length: 1638400 exceeds maximum: 16384

How can I debug this better? Is this frame legth set somewhere? Who can control if it exceeds.

[carl-mastrangelo]

You could try turning on frame logging:

https://gist.github.com/carl-mastrangelo/49f6d6a8ff29200fcb7d9e25e473b2d0

It's odd that netty would break the frame length. Also, would it be possible for you to try with v1.9.1 ?

[mfirry]

I'll try and upgrade this week.

[ejona86]

1638400 is suspicious, as it is exactly 100 * 16384. This error should not really ever happen, as the frame size is negotiated ahead-of-time. Generally this is not a bug on the side you see the error. What software was this connected to (if client, what server; if server, what client)? Also, were there any HTTP/2 proxies involved?

[mfirry]

Setup was mentioned here #3684 (comment)

Server is a Scala app running Marathon-lb (a wrapper around HAProxy).
Client runs on Tomcat.

[ejona86]

@mfirry, I'd suggest filing a bug against HAProxy (assuming it is in HTTP/2 mode; not TCP proxying).

[mfirry]

I don't think HAProxy supports HTTP2 yet

[ejona86]

HAProxy seems to support HTTP/2 since 1.8.

@mfirry, so you're doing TCP proxying via HAProxy with gRPC Java on client-side and server-side? I see that server-side is plain-text; is the client plain-text as well? (As in you aren't doing TLS stripping in the proxy.)

[mfirry]

Not doing tls right now

[mfirry]

Yes, grpc-java both sides

[ejona86]

Hmmm... There's not much to go on. We've seen this error several times, but it's yet to be gRPC's fault. I've seen broken HTTP/2 implementations and I've seen the client connecting to the wrong port/protocol.

Since it is plain-text, that means there are fewer verifications. 1638400 = 0x190000 which could maybe be trash. Nothing jumps out.

Frame logging as @carl-mastrangelo mentioned or a reproduction case would be helpful. Or sometimes simply knowing how you reproduce it can be enough.

[mfirry]

Thanks a lot for your time. I’ll see if I can reproduce it and let you know.

[ejona86]

@mfirry, sorry that we couldn't figure this out more. If you end up seeing more you can report back on this issue. Closing for the moment since there's not much for us to do.