Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graceful shutdown #58

Closed
pkieltyka opened this issue Jul 4, 2016 · 7 comments
Closed

Graceful shutdown #58

pkieltyka opened this issue Jul 4, 2016 · 7 comments

Comments

@pkieltyka
Copy link
Member

pkieltyka commented Jul 4, 2016

I'd like to explore ideas in how to gracefully shutdown a running http server. Now that go 1.7 supports context, we can use the request context to signal to all running requests the intent to shutdown the server and finish up their work because we're going to be exiting the program.

Initially the thought was to have a single parent server context where each routing context was derived, and if the parent was shutting down, cancelling the server context would signal to stop each child context. The problem here, is cancelling isn't the right thing to do, and it's actually the only option thats built into context.Context itself (see: golang/go#14660 (comment)).

On second thought, we could include a ShutdownCh on the request context, that would enhance each request to signal whenever the server is intending to finish up its work and stop processing. This is the best idea I have so far.

@pkieltyka
Copy link
Member Author

pkieltyka commented Jul 4, 2016

it's also likely someone will solve this outside of chi using Go 1.7's request context and it will "just work" with chi since we only depend on the stdlib net/http and context.

@VojtechVitek
Copy link
Contributor

We should discuss the Graceful Shutdown workflow:

On SIGINT/SIGTERM (or any other registered signal):

1a. Close listening socket and refuse any incoming connections.
1b. or .. Start responding with HTTP 503.
2. Disable keep-alive connections (how about http2?).
3. Start time.After(softTimeout) to let active requests finish.
4. When softTimeout expires, close the root context. Start time.After(hardTimeout).
5. Wait for all active requests to finish (middleware adding/removing value from channel or atomic.Uint64)
6. When hardTimeout expires, exit program.

  • softTimeout lets active requests finish their work without closing the context chain

I think all of the above except for 1a and 2 can be solved by middleware.

@kanocz
Copy link
Contributor

kanocz commented Aug 15, 2016

"1a" variant is much better (allows to use SO_REUSEPORT and start second instance)
P.S.: it's possible just to use https://github.com/tylerb/graceful (or get some inspiration there)

@cyx
Copy link
Contributor

cyx commented Sep 27, 2016

Yep, right now we're using goji/graceful, and it appears to work, so I'm curious what @pkieltyka is thinking re: his comment that what we're doing wouldn't work. Maybe subtle race conditions?

@pkieltyka
Copy link
Member Author

@cyx btw, this is what I was thinking https://github.com/goware/valve/blob/master/_example/main.go

its agnostic to chi, but of course works well with it

@cyx
Copy link
Contributor

cyx commented Oct 7, 2016

@pkieltyka nice, I see what you mean now -- you're very precise in the teardown process so I guess the way we're doing it now will be hand-waivy about things at best.

Excited to see something first class in chi?

@pkieltyka
Copy link
Member Author

pkieltyka commented Oct 7, 2016

@cyx yea exactly, it's important to preempt certain code blocks properly and that is where the valve package comes in. I'm going to update it's README soon, but the thought is its designed as an extension to context.Context, that adds an additional valve signal that when stopped will tell non-preemptive code blocks to stop processing and finish up, aka, the taps are turning off, stop what you’re doing.

it's already first-class to chi since it just relies on context.Context. It's not specific to just http servers, its useful for any kind of server. I have however moved the valve package to https://github.com/pressly/valve

I thought to write something like github.com/tylerb/graceful from scratch and having it right as a part of chi/graceful - but there is no point, I'd just be ripping their code for what does a very fine job. The missing part was signaling the shutdown and having proper code to let non-preemptable code finish. At this present time, I recommend the approach https://github.com/pressly/chi/blob/master/_examples/graceful/main.go but I'm always open to feedback and future ideas. Let me know how it works for all :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants