Failure Detectors

Failure detectors are central to the operation of the gossip protocol and they provide the critical information as to whether endpoints are alive, dead, or merely a little flaky. Rather than prescribing a particular method of failure detection, this functionality is provided via an interface which provides a generic api for interacting with time based failure detectors.

FailureDetectorFactory

Failure detectors are endpoint based and a factory pattern is used to create these instances as needed.

Provided failure detectors

Several failure detectors are provided with the system and provide varying implementations of failure detectors.

• AdaptiveFailureDetector An adaptive accural failure detector based on the paper: "A New Adaptive Accrual Failure Detector for Dependable Distributed Systems" by Benjamin Satzger, Andreas Pietzowski, Wolfgang Trumler, Theo Ungerer

• PhiAccrualFailureDetector Based on the paper, "The phi Accrual Failure Detector", by Naohiro Hayashibara, Xavier Defago, Rami Yared, and Takuya Katayama

• SimpleTimeoutFailureDector Just like it sounds

Of these, the AdaptiveFailureDetector is the best bet. It proves to be as accurate as the PhiAccrualFailureDetector, and is far less expensive. The PhiAccrualFailureDetector sounds cool, but requires signifcantly more computation and space. The SimpleTimeoutFailureDector is something you should only use if you're completely insane.

See the source for construction details.

How do I determine the parameters?

That's a really good question and I'll limit the discussion to the AdaptiveFailureDetector. First, I'd strongly suggest you read the paper as it provides good background as to the theory of operation. But briefly, this failure detector uses heartbeat intervals and constructs a model. When we periodically sample the detector, it uses this model to determine the probability that the endpoint is, in fact, dead at the sample time.

The constructor of the failure detector is the same for its factory

AdaptiveFailureDetector(double convictionThreshold,
                        int windowSize, double scale,
                        long expectedSampleInterval,
                        int initialSamples,
                        double minimumInterval)

The conviction threshold is the probability level the detector has to meet for conviction. This is a number between 0 and 1 (probability) where 0 means it has no clue and 1.0 means that it must be 100% sure that it is a dead parrot. Obviously, these two extremes are useless and so the art comes in setting this value high enough that you don't get false positives and low enough that you catch real failures quickly. I run this about 0.9, which seem to work well.

The detector works by keeping a sliding window of heartbeat intervals and the window size parameter is used to control this. The more samples in the window, the better the system model will be of the endpoint. However, this takes space and thus setting it too high is unnecessarily wasteful. I've been having good luck with 100 samples.

The scale factor of the detector is a shameless slop factor that tries to mitigate some of the delays that are inherent in the system. Set it as close to 1.0 as you can. The expected sample interval is used to "prime" the failure detector. Accrual failure detectors work by developing a statistical model of the endpoint. By its very nature, this is an adaptive process and if there is no model then the detector cannot make rational decisions. This problem has vexed any number of systems that try to use accrual failure detectors. Until the detector has enough datapoints, the detector makes an awful lot of false positives to endpoints that are still alive.

To solve this problem, we first "prime" the detector with a number of fabricated samples to give the detector something to work with. This is essentially assuming the endpoint has been performing properly and then letting the system discover the true model over time. We do this in this detector by supplying the expected interval time (in milliseconds) and the number of samples to prime the window with. This value should be set to some multiple (3 or so) of the gossip interval so that initial churn does not kill off live endpoints prematurely.

Finally, the minimum interval parameter provides us with another mechanism to deal with noisy and spiky data. This number represents what should be the minimum sample interval and measured intervals that fall below this value are discarded as noise. This value is usually set to the expected gossip interval in the system.