net/http: add built-in graceful shutdown support to Server #4674

Closed
gopherbot opened this Issue Jan 18, 2013 · 43 comments

Projects

None yet
@gopherbot

by patrick.allen.higgins:

http.Server only offers flavors of Serve() without a way to shut them down. Closing the
listener should make the server stop, but there seems to be a race in TLS servers where
this does not always work.

Further, in-progress requests cannot complete before exiting without some kind of
wrapper for net.Listener and net.Conn which includes synchronization between
Listener.Accept() and Conn.Close().

See https://groups.google.com/d/topic/golang-nuts/i4_kPK-rVxI/discussion for some
details.

A use-case for this is zero-downtime restarts: a master process opens a listener and
then spawns a slave, passing it the listener. When the master receives a SIGHUP, it
spawns a new slave. When the slave starts servicing requests, it signals the master and
the master signals the old slave to shutdown. The old slave must shutdown gracefully to
prevent service interruption.
@alberts
Contributor
alberts commented Jan 30, 2013

Comment 1:

Sounds familiar... http://tomayko.com/writings/unicorn-is-unix
@adg
Contributor
adg commented Jan 30, 2013

Comment 2:

You mention a few separate issues.
1. net/http doesn't offer a convenience function for net.Listen followed by http.Serve.
2. closing the underlying net.Listener doesn't shut down the http.Serve loop (because of
a race condition?)
3. there's no mechanism to gracefully shut down an http.Serve loop, waiting for any
in-flight requests to complete
I think the second point is a priority. The first doesn't seem very interesting to me.
The third point is a feature that we should explore further.
This issue should be about the third point only. Can you please file issues for the
other two points? I would especially like as much information as you have about the
second point.
Thanks

Status changed to Accepted.

@adg
Contributor
adg commented Jan 30, 2013

Comment 3:

Labels changed: added priority-later, go1.1maybe, packagechange, removed priority-triage, go1.1.

@gopherbot

Comment 4 by patrick.allen.higgins:

I will try to boil down a test case for the second point. I didn't file an issue for it
because fixing the third point would likely fix the second as well.
@gopherbot

Comment 5 by patrick.allen.higgins:

I created issue #4737, which I believe is related to the second point.
@gopherbot

Comment 6 by patrick.allen.higgins:

My proposal for this issue is to put an optional *sync.WaitGroup into http.Server and
modify (*Server).Serve to call Add(1) and Done() around the creation of servicing
goroutines. That would accomplish what my net.Listener wrappers are doing.
@robpike
Contributor
robpike commented Mar 7, 2013

Comment 7:

Labels changed: removed go1.1maybe.

@rsc
Contributor
rsc commented Jul 30, 2013

Comment 8:

Labels changed: added go1.2.

@rsc
Contributor
rsc commented Jul 30, 2013

Comment 9:

Labels changed: added feature.

@robpike
Contributor
robpike commented Aug 29, 2013

Comment 10:

Won't make it in time for 1.2.

Labels changed: removed go1.2.

@daaku
daaku commented Sep 9, 2013

Comment 11:

It would be nice to have a Server.CloseIdleConnections (to kill keep-alive connections)
much like we have Transport.CloseIdleConnections for the Client. There isn't a way for
an external library to do this since the state is not exposed.
@rsc
Contributor
rsc commented Nov 27, 2013

Comment 12:

Labels changed: added go1.3maybe.

@rsc
Contributor
rsc commented Nov 27, 2013

Comment 13:

Labels changed: removed feature.

@rsc
Contributor
rsc commented Dec 4, 2013

Comment 14:

Labels changed: added release-none, removed go1.3maybe.

@rsc
Contributor
rsc commented Dec 4, 2013

Comment 15:

Labels changed: added repo-main.

@gopherbot

Comment 16 by r@rcrowley.org:

After making some headway on this problem outside the standard library I decided to take
a stab at it in net/http itself.
https://golang.org/cl/67730046/
@bradfitz
Member

Comment 17:

Sent out https://golang.org/cl/69260044/ for discussion. It should be
sufficient to implement shutdown outside of net/http, with varying levels of
gracefulness.

Labels changed: added release-go1.3maybe, removed priority-later, release-none.

@bradfitz
Member

Comment 18:

Also sent out https://golang.org/cl/69670043 for disabling keep-alives (sending
"Connection: close" in Handlers)
@bradfitz
Member

Comment 19:

This issue was updated by revision 67b8bf3.

This allows for all sorts of graceful shutdown policies,
without picking a policy (e.g. lameduck period) and without
adding lots of locking to the server core. That policy and
locking can be implemented outside of net/http now.
LGTM=adg
R=golang-codereviews, josharian, r, adg, dvyukov
CC=golang-codereviews
https://golang.org/cl/69260044
@bradfitz
Member

Comment 20:

This issue was updated by revision 916682e.

LGTM=adg, josharian
R=adg, josharian, r
CC=golang-codereviews
https://golang.org/cl/69670043
@bradfitz
Member
bradfitz commented Mar 2, 2014

Comment 21:

This issue was updated by revision 7124ee5.

LGTM=bradfitz
R=bradfitz
CC=golang-codereviews
https://golang.org/cl/70410044
Committer: Brad Fitzpatrick 
@bradfitz
Member
bradfitz commented Mar 5, 2014

Comment 22:

Status update: Go 1.3 will have all the necessary net/http changes (Server.ConnState and
Server.SetKeepAlivesEnabled) for other people to implement graceful shutdown properly,
outside the net/http package.
But because there are various policy questions of how quickly and how politely to shut
down a server, we're going to refrain from unilaterally making that policy decision now
(in the form of e.g. a Server.Close method) and instead make people pick their own
policy for now.
It's possible in the future (after some experience using different shutdown strategies
in Go 1.3) we might see the common patterns and add a convenience shutdown/close method,
perhaps with a timeout or options, once we know what the set of options are.
So punting this to Go 1.4, even though most of the work will already be in Go 1.3.

Labels changed: added release-go1.4, removed release-go1.3maybe.

@rsc
Contributor
rsc commented Sep 16, 2014

Comment 23:

This didn't happen for 1.4. Maybe 1.5.

Labels changed: added release-go1.5, removed release-go1.4.

@bradfitz bradfitz modified the milestone: Go1.5 Dec 16, 2014
@bradfitz bradfitz removed the release-go1.5 label Dec 16, 2014
@bas-vk bas-vk referenced this issue in ethereum/go-ethereum Apr 17, 2015
Merged

admin.StopRPC added to console #734

@bradfitz
Member

There have been no proposals during the Go 1.5 dev cycle. I trust people are still happy with the ConnState hooks added in Go 1.3, as evidenced by its use in:

I'm changing this to Milestone "Unplanned", but if somebody wants to see this in Go 1.6, please make a proposal before ~Feb 2016.

@bradfitz bradfitz modified the milestone: Unplanned, Go1.5 Apr 30, 2015
@ghost
ghost commented Jul 16, 2015

How do I make a proposal? I'd love to have the ability to call Close() on an http server object. It's weird that I can't do it already and have to use a third party package to do that.

@bradfitz bradfitz changed the title from net/http: allow graceful shutdown to net/http: add built-in graceful shutdown support to Server Dec 23, 2015
@FiloSottile
Contributor

Until #9478 is fixed or something like CloseIdleConnections is added, the ConnState hook is of little use, since a graceful restart will still have to wait for all idle connections to timeout or be used once (and be shut by SetKeepAlivesEnabled). Correct?

@bradfitz
Member
bradfitz commented Aug 3, 2016

Why do you have to wait for idle connections? If they're idle, just close them or restart your process.

@FiloSottile
Contributor
@bradfitz
Member
bradfitz commented Aug 21, 2016 edited

(Deleted some +1 comments per our comment policy)

Please click the emoji thumbs-up at the top of threads to express your desires. Or write a proposal & prototype.

@leakingtapan

Any progress on this issue?

@bradfitz bradfitz modified the milestone: Go1.9, Unplanned Oct 29, 2016
@bradfitz bradfitz self-assigned this Oct 29, 2016
@bradfitz
Member

No, but I think a plan has started to form. It'll be a blocking:

func (s *Server) Shutdown(context.Context) error {}
func (s *Server) Close() error {}

Where the first one is graceful and takes a context (so you can abort early if it's taking too long), and the second one is forced and immediate. It would only return an error to fit the normal io.Closer signature, but would basically always return nil, even if it broke a bunch of in-use connections.

The Go 1.8 dev tree closes Sunday night, so I'll mark this as Go 1.9.

@dominikh
Contributor

Is there actually any point in implementing io.Closer? I don't think that an HTTP server really falls into that category, or that one would want to use it interchangeably with other io.Closers

@cespare
Contributor
cespare commented Oct 29, 2016

@dominikh Given that Listener.Close returns an error, wouldn't you want to pass that on?

@dominikh
Contributor

@cespare Good point. I was thrown off by Brad saying that it would always return nil.

@bradfitz
Member

"but would basically always return nil" :-)

Yeah, passing on the net.Listener(s)' return value makes sense.

@gopherbot

CL https://golang.org/cl/32329 mentions this issue.

@bradfitz bradfitz modified the milestone: Go1.8, Go1.9 Oct 30, 2016
@bradfitz
Member

I managed to implement this in a few hours before the freeze, and people have been wanting this forever, so Go 1.8 it is.

@gopherbot

CL https://golang.org/cl/32412 mentions this issue.

@gopherbot gopherbot pushed a commit to golang/net that referenced this issue Oct 30, 2016
@bradfitz bradfitz http2: add support for graceful shutdown of Server
This adds support for gracefully shutting down the Server, including
sending a GOAWAY frame to clients and shutting down when things are
idle. For now this support is only when integrated with the standard
library's Server.

A future change could export some of this.

Updates golang/go#4674

Change-Id: I78cd4f58ca529bf9d149054f929d9089e7685875
Reviewed-on: https://go-review.googlesource.com/32412
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
73058b0
@gopherbot gopherbot pushed a commit that closed this issue Nov 1, 2016
@bradfitz bradfitz net/http: add Server.Close & Server.Shutdown for forced & graceful shโ€ฆ
โ€ฆutdown

Also updates x/net/http2 to git rev 541150 for:

   http2: add support for graceful shutdown of Server
   https://golang.org/cl/32412

   http2: make http2.Server access http1's Server via an interface check
   https://golang.org/cl/32417

Fixes #4674
Fixes #9478

Change-Id: I8021a18dee0ef2fe3946ac1776d2b10d3d429052
Reviewed-on: https://go-review.googlesource.com/32329
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
53fc330
@gopherbot gopherbot closed this in 53fc330 Nov 1, 2016
@joeybloggs

I may be missing something, but it looks like hijacked connections are just removed from the tracked connections and if I'm not mistaken WebSockets hijack the connection; so my question is, with this change how would one ensure graceful shutdown of connections including WebSockets?

It appears that they would be interrupted/closed after all normal http idle connections are completed and the process ends.

I've only been able to do it using a custom listener that keeps track of the connections created and closed, ConnState for keeping track of idle KeepAlive connections all bundled in a library, kms, that exposes a ShutdownInitialized channel that WebSockets, and anything else, can use to gracefully shut themselves down with.

@bradfitz
Member
bradfitz commented Nov 1, 2016
@joeybloggs

@bradfitz out of curiosity along with the current implementation, why not provide a ShutdownInitialized channel that is accessible via a function from the *Server(or some other means like http.Request which would be more convenient) and then keep track of all of the connections via a WaitGroup, by wrapping the Server's listener and then hijacked connections would have the option of gracefully shutting down by using the ShutDownInitialized.

then server.Shutdown() can wait using the WaitGroup(), context and Timeout.

It just seems like having graceful shutdown built into net/http that doesn't support some pretty common cases seems incomplete/wrong somehow.

@bradfitz
Member
bradfitz commented Nov 1, 2016

Let's comment on new issue(s) instead of a closed issue. The new issue(s) can reference this one, and then they'll be cross-linked. Also, it's more constructive (at least for me) if you explain the problematic or desired behavior, and not the implementation. The implementation is generally straight-forward given the requirements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment