Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: No way in HTTP2 for http.Handler to request a client to close its connection #20977

Closed
smarterclayton opened this issue Jul 11, 2017 · 17 comments

Comments

Projects
None yet
8 participants
@smarterclayton
Copy link

commented Jul 11, 2017

What version of Go are you using (go version)?

1.8.1

What operating system and processor architecture are you using (go env)?

linux/amd64, darwin/amd64

What did you do?

The golang net/http HTTP1 server will close a client connection if a Connection: close header is written to the http.ResponseWriter. In HTTP/2, this will be passed directly to clients (resulting in an HTTP2 protocol error since that header is explicitly disallowed via the spec).

The reason the server was sending that header was to request (under excessive load) that Kubernetes API clients should "go away" and reconnect, to ensure that if one particular API server becomes overloaded that it can shed connections and let the intervening load balancer have another opportunity to balance out the load. Currently the kubernetes clients are very good at not needing to reconnect, and while the server knows that it is overloaded, the client has no way of knowing that another api server exists. Even if the client were to interpret the specific error response (429) as closing the connection, it can't effectively close the specific connection pointing to the overloaded server, given the API of the golang client transport. Finally, this requires every language client to implement that logic at a level above the HTTP2 transport, rather than as part of the normal HTTP2 behavior.

This is a request to have a mechanism in net/http for HTTP2 to request the HTTP2 server close a particular client connection, whether via receiving a Connection: close or some other mechanism. Without casting to private interfaces, it does not appear possible (or if it were, safe, considering the other protocol logic) to request a GOAWAY frame be sent from the server. This would allow an HTTP2 server to force a client to reestablish its HTTP2 connection, which would allow higher level proxies and balancers to appropriately route / reject that connection.

@bradfitz bradfitz changed the title http: No way in HTTP2 for http.Handler to request a client to close its connection net/http: No way in HTTP2 for http.Handler to request a client to close its connection Jul 11, 2017

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jul 11, 2017

@bradfitz bradfitz added this to the Go1.10 milestone Jul 11, 2017

@tombergan

This comment has been minimized.

Copy link
Contributor

commented Jul 12, 2017

@smarterclayton Do you use HTTP/2 via net/http or do you use x/net/http2 directly? net/http is currently very centered around HTTP/1 and doesn't have a real notion of a "connection".

Ideas:

  1. Add x/net/http2.ServeConn, with Serve and Shutdown methods. A handler could get at the request's ServeConn by extracting a value from the request's Context. This feels kind of ugly and doesn't backport nicely to net/http.

  2. Define an H2-specific handler type that exposes http2.ServeConn directly. This could have other advantages, e.g., the H2-specific handler could use a concrete ResponseWriter type that has a Push method, so users could call Push without a type cast. This feels less ugly, but doesn't backport nicely to net/http.

  3. Define interface { GoAway() } which is implemented by the HTTP/2 ResponseWriter. The semantics are: GoAway ends the current response and gracefully shuts down the client connection. This interface could be defined in net/http. We are starting to collect interfaces like this (also see http.Pusher and http.CloseNotifier), which feels ugly.

  4. Similar to the above, but instead the interface is something like interface { SetCloseConnectionAfterResponse() }. The semantics are: the method must be called before WriteHeader. In HTTP/1, the response will have Connection: close. In HTTP/2, a GOAWAY will be sent after the response. This works identically for H1 and H2, which is nice, but I can't think of a non-ugly method name, which is not nice.

  5. Reuse Connection: close in HTTP/2. I don't like this idea. This could result in surprising behavior, for example, for people who write H2 -> H1 proxies and forget to remove hop-by-hop headers when generating the proxied response.

@bradfitz, any thoughts?

Somewhat relatedly, the godoc for net/http.Handler says the following, which is actually not true for HTTP/2. (In H2, we send a RST_STREAM on panic.)

If ServeHTTP panics, the server ... recovers the panic, logs a stack trace to the server error log, and hangs up the connection.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jul 12, 2017

What do you actually want to do?

(1) hard-close the TCP connection and break all other streams currently in-flight?
(2) just send a GOAWAY, gracefully finishing the current active requests?
(3) send a REFUSED_STREAM, so the client retries it elsewhere?

I don't want to overload Connection: close for sure.

Another possibility is magic panic values, since net/http has historically (for better or worse) defined that panic is caught by the server and mapped into a server error. We can say that if the panic value implements a certain interface (say, interface { SendGoAway() } or interface { SendRefusedStream() } etc), then we do that. Then it also doesn't matter type-wise if the user uses the bundled x/net/http2 in std but uses the panic values from x/net/http2.

But let's figure out the goals before we design.

@smarterclayton

This comment has been minimized.

Copy link
Author

commented Jul 17, 2017

Kubernetes may be a bit unusual in this regard in that it has large numbers of very long lived connections from large numbers of stable clients (infrastructure) that process large numbers of individual requests. To avoid wasting CPU on TLS setup, we work very hard to avoid closing connections unless necessary. However, occasionally a backend server is overloaded and wishes to shed load. To do that, we send 429. However, with lots of stable clients (including browsers), the 429 doesn't actually shed load from that particular backend server, because clients continue to use that connection as long as it is open. This leads to rolling updates sloshing large chunks of load around, without balancing them (because the moment the server goes down, the clients reconnect because of the high rate of requests).

For this use case, for HTTP1, closing the TCP connection works well because intervening HTTP and TCP proxies handle that very well. Since that's one request in flight at a time, closing is acceptable because it does not impact other clients.

For HTTP2, given the lack of advanced HTTP2 support in most of the proxies in common use, closing the TCP connection is the only mechanism that would move load around. We would prefer it be graceful. So that sounds like 2 above (we want the client to GOAWAY and to hit the load balancer again). But if REFUSED_STREAM would signal a reasonable HTTP2 client to create a new connection, that would be desirable.

@tombergan

This comment has been minimized.

Copy link
Contributor

commented Jul 20, 2017

(3) send a REFUSED_STREAM, so the client retries it elsewhere?
But if REFUSED_STREAM would signal a reasonable HTTP2 client to create a new connection, that would be desirable.

Servers cannot rely on REFUSED_STREAM to mean "try again on a different connection". That is not what the HTTP/2 spec says and it's not what is implemented by Chrome or Firefox. See code links here.

I think GOAWAY is the right mechanism for this.

@bradfitz bradfitz modified the milestones: Go1.10, Go1.11 Nov 17, 2017

@bradfitz bradfitz assigned bradfitz and tombergan and unassigned bradfitz Nov 17, 2017

@odeke-em

This comment has been minimized.

Copy link
Member

commented Mar 5, 2018

/cc @rs

@kelwang

This comment has been minimized.

Copy link

commented May 4, 2018

A quick solution is to have an exported version http2errClientConnClosed,
so we can just compare and ignore it.

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 4, 2018

@kelwang, I think you're confusing this issue with a different one.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jun 28, 2018

It turns out that Go's HTTP/2 server was improperly allowing Handlers to send a "Connection: close" response header, which is a protocol violation.

I'll kill two birds with one stone and while fixing that, also fix this bug and make the "Connection: close" also mean to send a GOAWAY, like we do for HTTP/1.x.

@gopherbot

This comment has been minimized.

Copy link

commented Jun 28, 2018

Change https://golang.org/cl/121415 mentions this issue: http2: make Server send GOAWAY if Handler sets "Connection: close" header

gopherbot pushed a commit to golang/net that referenced this issue Jun 28, 2018

http2: make Server send GOAWAY if Handler sets "Connection: close" he…
…ader

In Go's HTTP/1.x Server, a "Connection: close" response from a handler results
in the TCP connection being closed.

In HTTP/2, a "Connection" header is illegal and we weren't previously
handling it, generating malformed responses to clients. (This is one
of our violations listed in golang/go#25023)

There was also a feature request in golang/go#20977 for a way for
HTTP/2 handlers to close the connection after the response. Since we
already close the connection for "Connection: close" for HTTP/1.x, do
the same for HTTP/2 and map it to a graceful GOAWAY errcode=NO
response.

Updates golang/go#25023 (improves 8.1.2.2. Connection-Specific Header Fields)
Updates golang/go#20977 (fixes after vendor into std)

Change-Id: Iefb33ea73be616052533080c63b54ae679b1d154
Reviewed-on: https://go-review.googlesource.com/121415
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Jun 29, 2018

Change https://golang.org/cl/121695 mentions this issue: net/http: update bundled http2

@gopherbot gopherbot closed this in cdce824 Jun 29, 2018

@AjinkyaKher

This comment has been minimized.

Copy link

commented Jul 19, 2018

Guys what's the latest update on this? We just ran into this, & I spent the whole day in investigating why some clients are not closing connection, only to figure out HTTP2 clients are not receiving it at all.
What do I do server side to force HTTP2 clients to close the connection?

@AjinkyaKher

This comment has been minimized.

Copy link

commented Jul 19, 2018

I'm using ASP .NET server side btw.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jul 19, 2018

Send response header Connection: close.

@AjinkyaKher

This comment has been minimized.

Copy link

commented Jul 19, 2018

That doesn't work right, we're already sending it. Http/2 clients do not receive this header, only Http/1 clients do. I thought that's what this whole thread is about, to cfigure out a way to close Http/2 connections.

@agnivade

This comment has been minimized.

Copy link
Member

commented Jul 19, 2018

Maybe a stupid question, but you are using the latest Go master version right ? The fix is obviously not available in 1.10, or even the 1.11beta1 I believe.

If you still see this issue in latest master, please feel free to raise a new issue with all the necessary details. Thank you.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jul 19, 2018

@AjinkyaKher, HTTP/2 clients are not supposed to see a Connection header. See golang/net@97aa3a5 for details. You need to be using a recent (unreleased) Go version to see this. Or wait a couple hours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.