Document how a Kubernetes-style liveness probe may be implemented

[meqif]

Due to a bug in rdkafka, sometimes consumers stop processing one or more assigned partitions. That sort of issue is not detectable through a consumer-wide liveness check because an individual consumer may keep making progress in some but not all assigned partitions.

In a system owned by me, I tried to leverage consumer rebalance listeners (via monkey patching) to introduce a liveness probe that is able to detect lack of progress in individual partition.

The core idea is that the progress in each partition is tracked through the timestamp of some specific files and a Kubernetes liveness probe checks their age. After some failures (defined in the probe), the Kubernetes pod is marked as unhealthy by the probe and automatically restarted, which fixes the issue.

In more detail:

• the consumer object refreshes a "partition liveness" file whenever it processes a new message
• a custom consumer rebalance listener reaps all existing "partition liveness" files on partition (re-) assignment
  • it would be possible to manage these files in a more targeted manner with the on_partitions_revoked handler but ordering could cause issues; the simpler, blunt approach I documented is simpler and works correctly in practice
• the Kubernetes liveness probe executes a small bash script that checks if any of the "partition liveness" files are older than some maximum age

This required adding some basic support for registering a consumer rebalance listener, which in the system I own is actually implemented through a monkey-patch.

/cc @jbdietrich

[jbdietrich]

@meqif sorry it's taken us so long to do a proper review of this. I think this is the current state of affairs:

We now have three open PRs that add liveness probe functionality (or at least document it):

• Add built-in health check functionality for systems like Kubernetes #308
• This PR
• Event based Kubernetes liveness probe #319

#319 seems like the best fleshed-out implementation, so we will standardize on that as a documented solution.

However, the general idea in this PR of exposing an API to respond to rebalances is valuable outside of the context of a K8s liveness probe. Would you be willing to extract that part of the PR for review?