Reduce Histogram Cardinality

[leoparente]

No description provided.

[leoparente]

PR Explanation:

Problem: Current Histogram implementation has a high cardinality due the way that datapoints are calculated. Moreover, the different datapoints almost every bucket do not allow to proper plot histogram over time.

Solutions:

First attempt: Use Opentelemetry Exponential Histogram - after investigating it, notice that there is no way to map exponential histogram to prometheus structure.. It is not supported by prometheus and it would not be good to do a implementation exclusively to Opentelemetry (without support to prometheus and json).

This PR attempt: Prometheus histogram high cardinality problem seems to be a common issue in the community. One custom implementation to that is provided by VictoriaMetrics

The solution — VictoriaMetrics histogram
We at VictoriaMetrics decided fixing these issues, went to drawing board, designed human-friendly easy-to-use Histogram and added it to the lightweight client library for exposing Prometheus-compatible metrics — github.com/VictoriaMetrics/metrics. The Histogram just works:
1- There is no need in thinking about bucket ranges and the number of buckets per histogram, since buckets are created on demand.
2- There is no need in worrying about high cardinality, since only buckets with non-zero values are exposed to Prometheus. Usually real-world values are located on quite small range, so they are covered by small number of histogram buckets.
3- There is no need in re-configuring buckets over time, since bucket configuration is static. This allows performing cross-histogram calculations.

It can be found in this article https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350

I've adapted the solution for our datasketches approach

[weyrick]

Interesting - seems like a good goal, my only concern is whether we know that existing data/graphs won't change in a significant way if this rolls out? I see you had to change the test to remove some answers, for example

[leoparente]

Interesting - seems like a good goal, my only concern is whether we know that existing data/graphs won't change in a significant way if this rolls out? I see you had to change the test to remove some answers, for example

Histogram Metric is only used yet by Netprobe Handler and both feature(Histogram and Netprobe) were released (4.2.0) as BETA:

New Features in BETA (interfaces may still change)
• Flow Support (SFLOW/Netflow/IPFIX). How To Configure and Policies Advanced.
• Netprobe support. Docs.
• Histogram Metric #526
• DNS Handler Version 2.0 - focus on dns transactions (docs soon)