Graceful shutdown

[pkieltyka]

I'd like to explore ideas in how to gracefully shutdown a running http server. Now that go 1.7 supports context, we can use the request context to signal to all running requests the intent to shutdown the server and finish up their work because we're going to be exiting the program.

Initially the thought was to have a single parent server context where each routing context was derived, and if the parent was shutting down, cancelling the server context would signal to stop each child context. The problem here, is cancelling isn't the right thing to do, and it's actually the only option thats built into context.Context itself (see: golang/go#14660 (comment)).

On second thought, we could include a ShutdownCh on the request context, that would enhance each request to signal whenever the server is intending to finish up its work and stop processing. This is the best idea I have so far.

[pkieltyka]

it's also likely someone will solve this outside of chi using Go 1.7's request context and it will "just work" with chi since we only depend on the stdlib net/http and context.

[VojtechVitek]

We should discuss the Graceful Shutdown workflow:

On SIGINT/SIGTERM (or any other registered signal):

1a. Close listening socket and refuse any incoming connections.
1b. or .. Start responding with HTTP 503.
2. Disable keep-alive connections (how about http2?).
3. Start time.After(softTimeout) to let active requests finish.
4. When softTimeout expires, close the root context. Start time.After(hardTimeout).
5. Wait for all active requests to finish (middleware adding/removing value from channel or atomic.Uint64)
6. When hardTimeout expires, exit program.

• softTimeout lets active requests finish their work without closing the context chain

I think all of the above except for 1a and 2 can be solved by middleware.

[kanocz]

"1a" variant is much better (allows to use SO_REUSEPORT and start second instance)
P.S.: it's possible just to use https://github.com/tylerb/graceful (or get some inspiration there)

[cyx]

Yep, right now we're using goji/graceful, and it appears to work, so I'm curious what @pkieltyka is thinking re: his comment that what we're doing wouldn't work. Maybe subtle race conditions?

[pkieltyka]

@cyx btw, this is what I was thinking https://github.com/goware/valve/blob/master/_example/main.go

its agnostic to chi, but of course works well with it

[cyx]

@pkieltyka nice, I see what you mean now -- you're very precise in the teardown process so I guess the way we're doing it now will be hand-waivy about things at best.

Excited to see something first class in chi?

[pkieltyka]

@cyx yea exactly, it's important to preempt certain code blocks properly and that is where the valve package comes in. I'm going to update it's README soon, but the thought is its designed as an extension to context.Context, that adds an additional valve signal that when stopped will tell non-preemptive code blocks to stop processing and finish up, aka, the taps are turning off, stop what you’re doing.

it's already first-class to chi since it just relies on context.Context. It's not specific to just http servers, its useful for any kind of server. I have however moved the valve package to https://github.com/pressly/valve

I thought to write something like github.com/tylerb/graceful from scratch and having it right as a part of chi/graceful - but there is no point, I'd just be ripping their code for what does a very fine job. The missing part was signaling the shutdown and having proper code to let non-preemptable code finish. At this present time, I recommend the approach https://github.com/pressly/chi/blob/master/_examples/graceful/main.go but I'm always open to feedback and future ideas. Let me know how it works for all :)