New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

At Least Once Delivery on Graceful Shutdown #367

Merged
merged 14 commits into from Dec 18, 2017

Conversation

Projects
None yet
3 participants
@pulltab
Member

pulltab commented Dec 13, 2017

The following implements at-least-once delivery semantics on graceful shutdown.

We achieve this by:

  • Requiring sources / filters to inject Shutdown events into downstream hopper queues when they can ensure that no more events will be placed.

  • Restricting source/filter shutdown such that each incident producer has emitted a Shutdown. This ensures that all possible events have been consumed from each producer.

pulltab added some commits Dec 13, 2017

Sources Now Shutdown Sinks, Filters
Towards at-least-once delivery - sources now propagate
shutdown to all downstream consumers.  This ensures that Shutdown
events flow in a controlled way from roots of the routing topology
through to leaf nodes.

Sinks and filters shutdown immediately upon receipt of a shutdown.
Thus, we have no guarentee all events are processed prior to shutdown in
cases when multiple sources feed into the same sinks and/or filters.
Sinks / Filters Shutdown After Incident Producers
The final step towards at least once delivery - sinks and filters now shutdown
after their incident producers each have sent a Shutdown event.  This
ensures that, in cases when multiple producers are inputing into a
sink/filter, that each has finished emitting other metric types.
Thus, no more events are expected and it is safe to shutdown.

To accomplish this we had to inform sinks/filters of the sources/filters
which generate data into their receiver. Facilitating this, we now build up
an adjacency matrix and use it to both populate senders for
sources/filters and producer metadata for filters/sinks.

@pulltab pulltab changed the title from WIP - Sources Now Shutdown Sinks, Filters to At Least Once Delivery on Shutdown Dec 15, 2017

@pulltab pulltab requested review from blt, dparton and gliush Dec 15, 2017

@pulltab pulltab changed the title from At Least Once Delivery on Shutdown to At Least Once Delivery on Graceful Shutdown Dec 15, 2017

Show outdated Hide outdated src/filter/mod.rs Outdated
Show outdated Hide outdated src/matrix.rs Outdated
@blt

This comment has been minimized.

Show comment
Hide comment
@blt

blt Dec 15, 2017

Collaborator

@pulltab suggest running clippy on this. Latest nightly / clippy build. There's some clones of double references and what not in the new code that it points out.

Collaborator

blt commented Dec 15, 2017

@pulltab suggest running clippy on this. Latest nightly / clippy build. There's some clones of double references and what not in the new code that it points out.

Show outdated Hide outdated src/bin/cernan.rs Outdated
Show outdated Hide outdated src/bin/cernan.rs Outdated
Show outdated Hide outdated src/bin/cernan.rs Outdated
Show outdated Hide outdated src/bin/cernan.rs Outdated

pulltab added some commits Dec 15, 2017

Fixes a Bug in Filter Sources
Change moves population of filter sources after forwards have been
populated.  This allows filters to distinguish between incident edges by
the lack of a sender.
Fixes File Server Shutdown Logic
Previous logic didn't distinguish between 0 system events and >0 system
events.  The net result was file servers shutting down after a single
iteration.
source_worker
.readiness
.set_readiness(mio::Ready::readable())
.expect("Oops!");
.expect("Failed to set readiness!");

This comment has been minimized.

@blt

blt Dec 18, 2017

Collaborator

Heh, that's a nicer error message than what I had.

@blt

blt Dec 18, 2017

Collaborator

Heh, that's a nicer error message than what I had.

// from each of its
// upstream sources/filters.
total_shutdowns += 1;
if total_shutdowns >= sources.len() {

This comment has been minimized.

@blt

blt Dec 18, 2017

Collaborator

So cool.

@blt

blt Dec 18, 2017

Collaborator

So cool.

Show outdated Hide outdated src/sink/native.rs Outdated
@blt

Boy I'm excited about this.

@blt

blt approved these changes Dec 18, 2017

@pulltab pulltab merged commit a4da52d into master Dec 18, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@pulltab pulltab deleted the at-least-once branch Dec 18, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment