-
Notifications
You must be signed in to change notification settings - Fork 625
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RST_STREAM with code 2 / system error -505 when using HTTP CONNECT proxy #2744
Comments
It's hard to tell what exactly is going wrong here. It looks like the http2 client should be able to handle getting the settings frame early, but clearly it isn't here for some reason. One thing I noticed is that the log you supplied includes debug logs from the C++ part of the http2 implementation, but not the JavaScript part of the implementation. I'm not sure what would cause that, but it would have some relevant information. One thing I want to note is that this problem shouldn't happen with TLS enabled, because that changes how the beginning of a connection is handled. |
If you mean the snippet that is visible in the ticket description, that log has been severely truncated, in the interest of making it easy to recognize other users searching for similar bugs in the future. As for you, you will be more interested in the attached zip file, which includes the complete log 😉. |
No, I'm talking about the log in the zip file. The Node http2 module has two major components: the C++ part that wraps nghttp2 and makes it interact with the Node runtime, and the JavaScript part that provides the public API. As far as I can tell, the log file you provided contains logs from the C++ part but not the JavaScript part. The clearest indication of this is that I see this line from the C++ part but not this line from the JavaScript part in the log. This is an issue because I would expect this other line from the JavaScript part to be a relevant indicator for this problem. I haven't dug into Node's built in debug logging, so I don't know what exactly would cause this discrepancy. |
Yeah, I assumed that much. By the way, this is not a blocker on our side. For testing purpose, patching the simple proxy server is enough, and for production use cases, I expect everyone would TLS the HTTP2/gRPC connection. I shared my findings mainly because I had already done the investigation that far, and at that point, it is easily reproducible, contrarily to most other cases of RST_STREAM/code 2 errors. But no pressure. |
I was missing Now, I can see Here's the command I'm using. Any flag/option I should turn on?
|
I got it. The problem is that node's HTTP response parser may take multiple packets ahead while filling its parsing buffer. This is exactly what happens if the server's first HTTP2 packet comes in before node is done parsing the This works locally. |
Problem description
When using an HTTP CONNECT proxy, it is possible for the server's first HTTP2 packet to reach the client before the latter sends its own first HTTP2 packet. If that happens, then the client terminates the connection (HTTP2 GOAWAY with code 2 and system error -505) and fail. Other gRPC implementations that we tested (Go, Java, and the Tokio third party implementation in Rust) don't have issue in that same scenario.
For example, in the capture below, we can see the server's first packet, at line 19, while the client's first packet, including the HTTP2 magic string is at line 21. Then, at line 37, we can see that the client side is initiating a GOAWAY, with code 2.
With gRPC and node's tracing enabled, we can see the following interesting messages (this is a truncated snippet; see the attached zip file for the complete log and corresponding pcap).
Reproduction steps
$ npm i
Environment
Additional context
I only have limited knowledge of the HTTP2 protocol internals, but as far as I understand, though unlikely, the fact that the server's first packet reaches the client before the client its first packet is technically a legit situation, and could also happen in some non-proxied scenarios (e.g. using TCP Fast Open).
Therefore, this might arguably be IMHO an oversight in the underlying HTTP2 implementation, rather than a bug in grpc-node's proxy support; still, I'm reporting this here since I wouldn't be able to debate this further without considerably more digging of my part.
Attached files
The text was updated successfully, but these errors were encountered: