feat: expose pending messages and processing rate #79
Conversation
Signed-off-by: Derek Wang <whynowy@gmail.com>
| package v1alpha1 | ||
|
|
||
| // VertexInstance is a wrapper of a vertex instance, which contains the vertex spec and the instance information such as hostname and replica index. | ||
| type VertexInstance struct { |
| } | ||
| } | ||
| pendingMessages.With(labels).Set(float64(total / num)) | ||
| } |
There was a problem hiding this comment.
should we emit these metrics to prometheus too?
There was a problem hiding this comment.
I assume you meant shall we display these metrics (they are already exposed/emmited) - This is the tricky part.
The pending_messages and processing_rate represent the vertex level metrics, not a pod, but we are exposing them from every individual pod, this will be a problem when displaying in the dashboard, that we should only display the metrics from one pod (or avg).
There was a problem hiding this comment.
This the reason of the discussion about should we only expose these metrics from 1 pod.
There was a problem hiding this comment.
we could expose from all pods, the numbers should be more or less the same.. we can let the user configure how they want to display avg/max/min etc?
There was a problem hiding this comment.
I agree with Vigith that we should expose from all the pods and let the user decide when they are displaying these metrics on the type of stat to be used. We just need to call this out specifically in our doc that these numbers are vertex level and should roughly be the same and the aggregation we need to use is just avg/max/min?
Signed-off-by: Derek Wang <whynowy@gmail.com>
* feat: expose pending messages and processing rate Signed-off-by: Derek Wang <whynowy@gmail.com>
* feat: expose pending messages and processing rate Signed-off-by: Derek Wang <whynowy@gmail.com> Signed-off-by: Yashash H L <yashash_hl@intuit.com>
Signed-off-by: Derek Wang whynowy@gmail.com
Processing rate and pending message are exposed in each vertex's prometheus, which will be used later for auto scaling calculation.