Processing Rate calculation optimization

[KeranYang]

Currently to calculate the processing rate of a vertex, we sum the count deltas between each of the timestamp interval pairs, which is an O(length of look back period) operation. If we assume no pod crash, then we can view the count-to-time relationship as a slop and quickly estimate the processing rate by only using the first and last data points. Rate = delta(count)/delta(t)

Considering that in PROD pod crash is a rare case, we can further assume within a single lookback period, the number of pod crash is constant. In that case, if we keep track of under which timestamp a pod crash was detected, then we just need 3 data points to calculate rates, first, last and the one when the pod crashed. This can be an O(3) operation.

This task is to investigate and experiment on this approach and see if it works in practice.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.