-
Notifications
You must be signed in to change notification settings - Fork 17.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
net/http: No way in HTTP2 for http.Handler to request a client to close its connection #20977
Comments
/cc @tombergan |
@smarterclayton Do you use HTTP/2 via net/http or do you use x/net/http2 directly? net/http is currently very centered around HTTP/1 and doesn't have a real notion of a "connection". Ideas:
@bradfitz, any thoughts? Somewhat relatedly, the godoc for net/http.Handler says the following, which is actually not true for HTTP/2. (In H2, we send a RST_STREAM on panic.)
|
What do you actually want to do? (1) hard-close the TCP connection and break all other streams currently in-flight? I don't want to overload Another possibility is magic panic values, since net/http has historically (for better or worse) defined that panic is caught by the server and mapped into a server error. We can say that if the panic value implements a certain interface (say, But let's figure out the goals before we design. |
Kubernetes may be a bit unusual in this regard in that it has large numbers of very long lived connections from large numbers of stable clients (infrastructure) that process large numbers of individual requests. To avoid wasting CPU on TLS setup, we work very hard to avoid closing connections unless necessary. However, occasionally a backend server is overloaded and wishes to shed load. To do that, we send 429. However, with lots of stable clients (including browsers), the 429 doesn't actually shed load from that particular backend server, because clients continue to use that connection as long as it is open. This leads to rolling updates sloshing large chunks of load around, without balancing them (because the moment the server goes down, the clients reconnect because of the high rate of requests). For this use case, for HTTP1, closing the TCP connection works well because intervening HTTP and TCP proxies handle that very well. Since that's one request in flight at a time, closing is acceptable because it does not impact other clients. For HTTP2, given the lack of advanced HTTP2 support in most of the proxies in common use, closing the TCP connection is the only mechanism that would move load around. We would prefer it be graceful. So that sounds like 2 above (we want the client to GOAWAY and to hit the load balancer again). But if REFUSED_STREAM would signal a reasonable HTTP2 client to create a new connection, that would be desirable. |
Servers cannot rely on REFUSED_STREAM to mean "try again on a different connection". That is not what the HTTP/2 spec says and it's not what is implemented by Chrome or Firefox. See code links here. I think GOAWAY is the right mechanism for this. |
/cc @rs |
A quick solution is to have an exported version http2errClientConnClosed, |
@kelwang, I think you're confusing this issue with a different one. |
It turns out that Go's HTTP/2 server was improperly allowing Handlers to send a "Connection: close" response header, which is a protocol violation. I'll kill two birds with one stone and while fixing that, also fix this bug and make the "Connection: close" also mean to send a GOAWAY, like we do for HTTP/1.x. |
Change https://golang.org/cl/121415 mentions this issue: |
…ader In Go's HTTP/1.x Server, a "Connection: close" response from a handler results in the TCP connection being closed. In HTTP/2, a "Connection" header is illegal and we weren't previously handling it, generating malformed responses to clients. (This is one of our violations listed in golang/go#25023) There was also a feature request in golang/go#20977 for a way for HTTP/2 handlers to close the connection after the response. Since we already close the connection for "Connection: close" for HTTP/1.x, do the same for HTTP/2 and map it to a graceful GOAWAY errcode=NO response. Updates golang/go#25023 (improves 8.1.2.2. Connection-Specific Header Fields) Updates golang/go#20977 (fixes after vendor into std) Change-Id: Iefb33ea73be616052533080c63b54ae679b1d154 Reviewed-on: https://go-review.googlesource.com/121415 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Change https://golang.org/cl/121695 mentions this issue: |
Guys what's the latest update on this? We just ran into this, & I spent the whole day in investigating why some clients are not closing connection, only to figure out HTTP2 clients are not receiving it at all. |
I'm using ASP .NET server side btw. |
Send response header |
That doesn't work right, we're already sending it. Http/2 clients do not receive this header, only Http/1 clients do. I thought that's what this whole thread is about, to cfigure out a way to close Http/2 connections. |
Maybe a stupid question, but you are using the latest Go master version right ? The fix is obviously not available in 1.10, or even the 1.11beta1 I believe. If you still see this issue in latest master, please feel free to raise a new issue with all the necessary details. Thank you. |
@AjinkyaKher, HTTP/2 clients are not supposed to see a Connection header. See golang/net@97aa3a5 for details. You need to be using a recent (unreleased) Go version to see this. Or wait a couple hours. |
What version of Go are you using (
go version
)?1.8.1
What operating system and processor architecture are you using (
go env
)?linux/amd64, darwin/amd64
What did you do?
The golang net/http HTTP1 server will close a client connection if a
Connection: close
header is written to thehttp.ResponseWriter
. In HTTP/2, this will be passed directly to clients (resulting in an HTTP2 protocol error since that header is explicitly disallowed via the spec).The reason the server was sending that header was to request (under excessive load) that Kubernetes API clients should "go away" and reconnect, to ensure that if one particular API server becomes overloaded that it can shed connections and let the intervening load balancer have another opportunity to balance out the load. Currently the kubernetes clients are very good at not needing to reconnect, and while the server knows that it is overloaded, the client has no way of knowing that another api server exists. Even if the client were to interpret the specific error response (429) as closing the connection, it can't effectively close the specific connection pointing to the overloaded server, given the API of the golang client transport. Finally, this requires every language client to implement that logic at a level above the HTTP2 transport, rather than as part of the normal HTTP2 behavior.
This is a request to have a mechanism in
net/http
for HTTP2 to request the HTTP2 server close a particular client connection, whether via receiving aConnection: close
or some other mechanism. Without casting to private interfaces, it does not appear possible (or if it were, safe, considering the other protocol logic) to request a GOAWAY frame be sent from the server. This would allow an HTTP2 server to force a client to reestablish its HTTP2 connection, which would allow higher level proxies and balancers to appropriately route / reject that connection.The text was updated successfully, but these errors were encountered: