Health / Liveness Endpoint

[mrueg]

It would be awesome if there were endpoints for health / liveness probes defined, (e.g. test if it can connect to k8s API) https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

[mustafaakin]

It's a good idea. However, testing whether it can connect to the API is not a good probe. If it cannot, it should be in crash-loop. However, readiness can be set after the pod performs the watch call to API. Watch is called once, then for labels, we get the objects. For clusters, that might not grant "read" permissions for all types (CRDs are not included by default) we ignore the error. So it gets trickier. What can be the solution?

[bygui86]

I suggest to include in the probes also a sort of connectivity check of all receivers.
For example, defining es, slack and stdout receivers:

• check if ES connectivity is fine and works
• check if slack push works
• check if stdout file (/dev/stdout) exists and is usable (permissions?)