Health checks, Kubernetes probes and Connect

[lkysow]

Currently the Consul Connect integration doesn't hook into Kubernetes health/liveliness probes nor do we provide a way to register Consul health checks. Thus when a Pod goes unhealthy we still route to it.

[pbusko]

We're experiencing the same issue here, in our use case TCP health check is not enough. Without integration with k8s readiness/liveness probes requests are frequently being routed to dead pods.

[lkysow]

Thanks Pavel, what do you think about us automatically hooking into the kube readiness probes?

[pbusko]

Thanks for your response @lkysow .

Readiness probe hook would be the best option imo. Liveness probe by it's nature doesn't guarantee that pod is ready to accept traffic (at least not always) while readiness probe hook could help to leverage native k8s capabilities without need to duplicate health checks (k8s + consul). Also in such case when not all services deployed in kubernetes are connect-enabled this approach allows to keep consistent way of health check definitions.

[jaredcurtis]

Hello,

We're investing using Consul connect for our service mesh and have seen this issue as well. We're heavy users of health checks for our existing consul service discovery. Does any workaround exist enable consul to check the health of a connect enabled service?

[lkysow]

The only workaround I can think of right now is to add a check yourself via consul's api for each connect pod.

[lkysow]

Health checks have now been released! We now sync Kubernetes probes to Consul. See https://www.consul.io/docs/k8s/connect/health. This was released in consul-helm v0.26.0.