Implement tonic-health

[akiradeveloper]

Implement tonic-health to support gPRC health-check.

The response would be a mix of status of the RaftCore and the RaftApp.

Protocol: https://github.com/grpc/grpc/blob/master/doc/health-checking.md

[akiradeveloper]

When this feature is ready, we can use this software to ask the aliveness: https://github.com/grpc-ecosystem/grpc-health-probe

[akiradeveloper]

I think this should be implemented in the same way as #184. The health-check service will be implemented as an overlay and can be in a separate crate.

The tonic-health says we should do so.

/// Creates a `HealthReporter` and a linked `HealthServer` pair. Together,
/// these types can be used to serve the gRPC Health Checking service.
///
/// A `HealthReporter` is used to update the state of gRPC services.
///
/// A `HealthServer` is a Tonic gRPC server for the `grpc.health.v1.Health`,
/// which can be added to a Tonic runtime using `add_service` on the runtime
/// builder.
pub fn health_reporter() -> (HealthReporter, HealthServer<impl Health>) {

[akiradeveloper]

• https://kubernetes.io/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/
• https://github.com/grpc-ecosystem/grpc-health-probe
• https://tomioka-shogorila.hatenablog.com/entry/2019/04/28/000540 (ja)

If this feature is implemented, we can be able to exploit the utilities from k8s and gRPC community.

[akiradeveloper]

Unhealthy condition for lol:

• App says so
• Core's log is blocked
• (maybe) tonic server says so

[akiradeveloper]

We need to add an interface like this.

impl RaftCore {
  pub fn add_health_reporter(reporter: HealthReporter) { self.reporter = Some(reporter); }
}

and set_serving and set_not_serving is called inside the core's code.

The HealthReporter may be passed to RaftApp as well. This way, HealthService can return current health that takes both core and app into account.

[akiradeveloper]

As lolraft implements in-process multi-raft. I think it became harder to define healthiness of the server.