Watcher System Metrics #8338
Watcher System Metrics #8338
Conversation
rosstimothy
requested review from
klizhentas,
r0mant and
russjones
as code owners
|
@fspmarshall @andrejtokarcik Can you two review? |
| Count: int64(bucket.GetCumulativeCount()), | ||
| UpperBound: bucket.GetUpperBound(), | ||
| }) | ||
| } | ||
| } | ||
| return out | ||
| } |
There was a problem hiding this comment.
It's not a big deal but your new additions reveal getComponentHistogram and getHistogram share most of the code, making their respective purpose/differences unnecessarily unclear. I suppose they could be nicely unified by means of a third function with the signature func(metric *dto.MetricFamily, hist *dto.Histogram) Histogram.
There was a problem hiding this comment.
Refactored to getHistogramWithFilter(metric *dto.MetricFamily, histogramFilter func(metrics []*dto.Metric) *dto.Histogram) Histogram to unite the shared code in one place and allow callers to provide filtering.
There was a problem hiding this comment.
Hm, this solution doesn't really seem to make the code more readable/succinct. Feel free to roll back to the original variant, as you prefer.
rosstimothy
force-pushed
the
tross/event-metrics
branch
2 times, most recently
from
ee32bb6
to
582a3b7
Compare
| Count: int64(bucket.GetCumulativeCount()), | ||
| UpperBound: bucket.GetUpperBound(), | ||
| }) | ||
| } | ||
| } | ||
| return out | ||
| } |
There was a problem hiding this comment.
Hm, this solution doesn't really seem to make the code more readable/succinct. Feel free to roll back to the original variant, as you prefer.
rosstimothy
force-pushed
the
tross/event-metrics
branch
from
582a3b7
to
2951df0
Compare
rosstimothy
force-pushed
the
tross/event-metrics
branch
from
2951df0
to
a58817b
Compare
* add event watcher prometheus metrics and a new tctl top tab to visualize them
* add event watcher prometheus metrics and a new tctl top tab to visualize them (cherry picked from commit fb0ab2b)
Purpose
Currently we have no insight into which events are being emitted, how often they are being emitted, and what the size of an event is. This makes it difficult to determine the source of high network utilization issues. By exposing metrics for the watcher system and consuming them on a new pane in
tctl topwe can get a real time view of the events being emitted.Implementation
-- record events in
WatchEventsstreaming GRPC method before weSendon the stream-- use
Event.Size()to estimate the amount of data being senttctl top