Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation:
Histograms are the Prometheus killer. They are big to send over the
network and are big to store in Prometheus.
Histograms' size is always an issue, but with Seastar, it becomes even trickier.
Typically, each shard would collect its histograms, so the overall data
is multiplied by the number of shards.
This series addresses the need to report quantile information like latency
without generating massive metrics reports.
A summary is a Prometheus metric type that holds a quantile summary (i.e.
p95, p99).
The downside of summaries is that they cannot be aggregated, which is
needed for a distributed system (i.e., calculate the p99 latency of a cluster).
The series adds four tools for Prometheus performance:
optional services. It is now possible to mark those metrics as
skip_when_empty and they will not be reported if they were never used.
histogram instead of per shard. For example, for multi-nodes quantile calculation,
we need a per-node histogram. It is now possible to mark a registered
metric for aggregation. The metrics layer will aggregate this
metric based on a list of labels. (Typically, this will be by shard,
but it could be any other combination of labels).
iteration.
V6
changed the name of the skip_when_empty variable to should_skip_when_empty to make the compiler happy.
V5
Main changes:
operator to mark that a metric should use skip_when_empty.
V4
Aggregation and skip_when_empty are now generalized in a sense that it's
a per metric configuration and can be used on any metric, not just
histogram.
The 'aggregate by' now accept labels instead of string which makes the API
cleaner.
The stringstream that is used inside Prometheus text reporting is now
reused instead of recreating on each iteration.
The aggregator object in Prometheus was rename and some comments were
added to make its use clearer.
V3
The main issue with the previous version was that it was too specific.
This version breaks the functionality and pushes some of the logic to
whoever uses the metrics layer.
Summaries are now just another kind of histogram, with no special magic
around them and no need for their own type.
A user can use any combination of labels for aggregation.
It's currently only applied to histograms, but we can do it with other
metrics if it will be helpful.
In the current implementation, to report a summary per shard and a single
histogram per node, the user of the metrics layer would register two metrics,
one for the summary and one for the histogram.