New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introducing Graceful Shutdown #339

Merged
merged 38 commits into from Dec 12, 2017

Conversation

Projects
None yet
4 participants
@pulltab
Member

pulltab commented Nov 10, 2017

Sources, sinks, and filters now shutdown gracefully and in a manner which fosters at least once delivery.

In response to a SIGINT or SIGTERM, the cernan main thread now:

  1. Gracefully terminates and joins all running sources. For sources with protocols enabled with ACKs, this will ensure downstream clients are aware when their intended source is no longer available. Graceful terminate is achieved by use of mio and its polling features to select between client traffic and so called SYSTEM events. At present there is only one supported SYSTEM event (SHUTDOWN).

  2. Gracefully terminates running filters and sinks, blocking until all have terminated. This is accomplished by injecting the new metric::event::Shutdown at the end of Hopper queue associated with the given consumer. Upon receipt of a metric::event::Shutdown, filters and sinks flush and terminate.

@pulltab pulltab requested review from blt and gliush Nov 10, 2017

@pulltab

This comment has been minimized.

Show comment
Hide comment
@pulltab

pulltab Nov 10, 2017

Member

Resolves #34

Member

pulltab commented Nov 10, 2017

Resolves #34

@blakebarnett

This comment has been minimized.

Show comment
Hide comment
@blakebarnett

blakebarnett Nov 10, 2017

Contributor

This is awesome, not proficient enough with rust to review properly but I can't wait to see it in action :)

Contributor

blakebarnett commented Nov 10, 2017

This is awesome, not proficient enough with rust to review properly but I can't wait to see it in action :)

Show outdated Hide outdated src/util.rs Outdated
Show outdated Hide outdated src/bin/cernan.rs Outdated
@blt

I'm still learning about Mio so the comments I have here so far are minimal. I'm going to start a test-bench for this branch.

Show outdated Hide outdated src/bin/cernan.rs Outdated
Show outdated Hide outdated src/source/internal.rs Outdated
@blt

This comment has been minimized.

Show comment
Hide comment
@blt

blt Nov 25, 2017

Collaborator

Hhhmmmm so this works fine in debug mode, but as soon as you compile in release-mode sending a SIGKILL will freeze things up.

Collaborator

blt commented Nov 25, 2017

Hhhmmmm so this works fine in debug mode, but as soon as you compile in release-mode sending a SIGKILL will freeze things up.

@blt

This comment has been minimized.

Show comment
Hide comment
@blt

blt Nov 25, 2017

Collaborator

No, no, I can trigger it in debug mode too. Never mind my last.

Collaborator

blt commented Nov 25, 2017

No, no, I can trigger it in debug mode too. Never mind my last.

Show outdated Hide outdated src/sink/prometheus.rs Outdated

@blt blt dismissed their stale review Dec 2, 2017

Boy it's been a while since that last review.

Show outdated Hide outdated src/source/graphite.rs Outdated
Show outdated Hide outdated src/source/native.rs Outdated
Show outdated Hide outdated src/sink/prometheus.rs Outdated
Show outdated Hide outdated src/sink/prometheus.rs Outdated
Show outdated Hide outdated src/sink/prometheus.rs Outdated
Show outdated Hide outdated src/sink/prometheus.rs Outdated
let host_port =

This comment has been minimized.

@blt

blt Dec 6, 2017

Collaborator

A tuple here should be able to coerce into the IP needed. Yeah, looks like (&str, u16) implements ToSocketAddrs.

@blt

blt Dec 6, 2017

Collaborator

A tuple here should be able to coerce into the IP needed. Yeah, looks like (&str, u16) implements ToSocketAddrs.

Show outdated Hide outdated src/sink/prometheus.rs Outdated
Show outdated Hide outdated src/source/file/file_server.rs Outdated
Show outdated Hide outdated src/source/flush.rs Outdated
Show outdated Hide outdated src/source/mod.rs Outdated
Show outdated Hide outdated src/source/internal.rs Outdated
Show outdated Hide outdated src/source/native.rs Outdated
let (len, _) = match socket.recv_from(&mut buf) {
Ok(r) => r,
Err(e) => panic!(format!(

This comment has been minimized.

@blt

blt Dec 6, 2017

Collaborator

In the event that we can't read from the socket here, what ought we to do? Is this a catastrophic failure – as indicated by the panic? – or something that can be recovered from? I'm not sure of the answer.

@blt

blt Dec 6, 2017

Collaborator

In the event that we can't read from the socket here, what ought we to do? Is this a catastrophic failure – as indicated by the panic? – or something that can be recovered from? I'm not sure of the answer.

This comment has been minimized.

@pulltab

pulltab Dec 6, 2017

Member

@blt blt referenced this pull request Dec 6, 2017

Closed

RFC: systemd-journald source #359

0 of 7 tasks complete
Show outdated Hide outdated src/thread.rs Outdated

pulltab added some commits Oct 24, 2017

Introducing Shutdown Event Type
Shutdown is emitted by the main thread of execution in response to
either SIGTERM or SIGINT.
Statsd, Internal converted to mio.
Graphite is in progress.  Pushing changes for help/feedback.

Brian L. Troutwine and others added some commits Nov 26, 2017

Brian L. Troutwine
Pin down the shutdown hang issue
I've added a 'shutdown' to the sink to allow for the controlled
shutdown of internal resources. The shutdown will hang for prometheus
sink because of the issue linked in the comments.

We're probably going to have to update prometheus sink HTTP server.
This also implies that, going forward, we'll have to be very careful
about testing shutdown functions for fully running sinks.

Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Re-introduce shutdown in the statsd source
I misunderstood what John was doing and broke his work. This commit
fixes the thing I broke: statsd source shutdown. What I'd misunderstood
was that the constants::SYSTEM token is _only_ readable when it's time
for a source to go the way of tears in the rain.

Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Quiet up the compiler during build
This branch was very chatty during build. It is no longer.

Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Cargo fmt run
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Re-introduce Sync + Send back into Handler constraints
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Re-introduce 'static for Handler bound in Server::new
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Quieten the build further
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Clear out dead comment
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Make Sink::shutdown own its structure
Previously shutdown took `&mut self` which meant that the Sink
storage could still be valid after shutdown. This stank in terms
of actually performing shutdown. Now the Sinks can be sure they
own everything on the way out.

Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Re-introduce write_text back into prometheus sink
This commit re-enables write_text in the prometheus sink. Had
to fiddle with the imports a touch but we're back on the road.

Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Pass a writable buffer into write_text
write_text no longer creates an http::Response but does require
a buffer to be passed into it, which it then fills up.

Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Cargo fmt run
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Some clippy suggestions integrated
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Fiddle with references
This commit moves some of the clones we were doing into reference
passes, saving on a handful of copies / increments of atomic counters.

Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Fix rebase issues
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Convert panic to warn
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Brian L. Troutwine
Remove excess externs / marked in review
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
Added TokenSlab
Replaces usage of HashMap to store Token -> Evented mappings.
Adding f64 Back to Prometheus Sink
Needed in latest master.
Brian L. Troutwine
Remove one extern crate mio
Signed-off-by: Brian L. Troutwine <blt@postmates.com>
@blt

blt approved these changes Dec 12, 2017

The day of glory has come.

@pulltab pulltab merged commit 6313149 into master Dec 12, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@blt blt deleted the ipc-shutdown branch Dec 12, 2017

@gliush

This comment has been minimized.

Show comment
Hide comment
@gliush

gliush Dec 13, 2017

Contributor

Cool!

Contributor

gliush commented Dec 13, 2017

Cool!

@blt blt referenced this pull request Jan 22, 2018

Closed

Handle Signals #34

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment