Add logging to remote watcher once heartbeat delays start to grow too large

[drewhk]

We should start logging at WARN levels once heartbeat delays grow over a threshold (for example half of acceptable heartbeat pause)

[fadyZohdy]

i started working on this task and from what i understand from the code this will be calculated in the failure detector with something like this and correct me if i am wrong

def isHeartbeatDelayed: Boolean = lastestInterval >= (acceptableHeartbeatPauseMillis / 2) 

so my question is should i just log the warning in the phaiAccrualDetector or should it be declared as a function in the FailureDetector trait and implemented in all the child classes and move the logging to the RemoteWatcher ?

[patriknw]

It's probably easiest to impl the logging in the concrete failure detector, e.g. backwards compatibility.