Interpolate for less variable consumer lag in time estimates in Kafka Lag Exporter

[seglo]

Today Kafka Lag Exporter simply polls on a small interval and extrapolates lag based on two measurements from the past two intervals. This works reasonably well with a steady consumption pattern, but if consuming “flattens out” during one of those intervals than it can lead to very inflated lag in time estimates.

A solution is to interpolate based on a sliding window of measurements of the latest offset measurement. We capture latest offsets for a partition every interval, then to calculate lag in time we lookup the interval our last read offset is in. We interpolate to find the timestamp for what the offset was between that measured interval, then use it to determine how far behind the current consumer is.