feat: Add tracking of heartbeat results and executions with stale heartbeat

[kagkarlsson]

Add tracking of heartbeat results and executions with stale heartbeat to enable acting on failing heartbeats.

• Configurable number of heartbeats before considered dead (minimum is 4)
  • New method schedulerBuilder.missedHeartbeatsLimit(<limit>)
• Increase default limit for missed heartbeats before considered dead from 4 to 6
• Track the number of failed heartbeats and make it available to users
  • New ExecutionContext method, ctx.getCurrentlyExecuting().getHeartbeatState().getFractionDead()
• New scheduler methods for checking for stale heartbeats
  • scheduler.getCurrentlyExecutingWithStaleHeartbeat()
  • scheduler.getCurrentlyExecuting().get(0).getHeartbeatState().getFractionDead()
• New SchedulerStatsEvents in StatsRegistry: FAILED_HEARTBEAT, FAILED_MULTIPLE_HEARTBEATS (replaces UNEXPECTED_ERROR for failed heartbeats)
• New example HeartbeatMonitoringMain

Fixes

• Improve missing heartbeat handling #402

Reminders

• Added/ran automated tests
• Update README and/or examples
• Ran mvn spotless:apply

[kagkarlsson]

@PiotrDuz @eyalschneider How about this for a start for #402 ? There is no callback-mechanism, but you could implement your own by monitoring scheduler.getCurrentlyExecutingWithStaleHeartbeat()

#433 might be of interest to you as well.

[kagkarlsson]

🎉 This issue has been resolved in v14.0.0 (Release Notes)