Too many per-partition metrics when aggregation is turned on #15811

travisdowns · 2023-12-20T15:27:31Z

Version & Environment

Redpanda version: 23.2

What went wrong?

When aggregate metrics is turned on, we still have too many metrics which have the partition label, which causes the number of metrics to be very high with high numbers of partitions.

For example, with 23k partitions, we have ~600k metrics on the /metrics endpoint and ~60k on the /public_metrics one.

What should have happened instead?

Fewer metrics.

How to reproduce the issue?

Get the endpoints

Additional information

This will be fixed by additionally aggregating some per-partition metrics.

The text was updated successfully, but these errors were encountered:

travisdowns · 2024-01-02T13:19:00Z

Related: https://github.com/redpanda-data/core-internal/issues/677

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the consensus object for two metrics: leader_for and reconfiguration changes in progress. This change enable aggregation on the partition label for these metrics if aggregation is turned on. Issue redpanda-data#15811. Issue redpanda-data/core-internal#677.

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the partition probe object. This change enable aggregation on the partition label for almost all metrics in the partition problem, with only two excluded: vectorized_cluster_partition_leader_id vectorized_cluster_partition_under_replicated_replicas Issue redpanda-data#15811. Issue redpanda-data/core-internal#677.

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the tx metrics. This change enable aggregation on the partition label for almost all three metrics in this probe. Issue redpanda-data#15811. Issue redpanda-data/core-internal#677.

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the partition probe object. This change enable aggregation on the partition label for almost all metrics in the partition problem, with only two excluded: vectorized_cluster_partition_leader_id vectorized_cluster_partition_under_replicated_replicas Issue redpanda-data#15811. Issue redpanda-data/core-internal#677.

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the tx metrics. This change enable aggregation on the partition label for almost all three metrics in this probe. Issue redpanda-data#15811. Issue redpanda-data/core-internal#677.

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the partition probe object. This change enable aggregation on the partition label for almost all metrics in the partition problem, with only two excluded: vectorized_cluster_partition_leader_id vectorized_cluster_partition_under_replicated_replicas Issue redpanda-data#15811. Issue redpanda-data/core-internal#677.

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the tx metrics. This change enable aggregation on the partition label for almost all three metrics in this probe. Issue redpanda-data#15811. Issue redpanda-data/core-internal#677.

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the consensus object for two metrics: leader_for and reconfiguration changes in progress. This change enable aggregation on the partition label for these metrics if aggregation is turned on. Issue redpanda-data#15811. Issue redpanda-data/core-internal#677. (cherry picked from commit 04b1535)

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the partition probe object. This change enable aggregation on the partition label for almost all metrics in the partition problem, with only two excluded: vectorized_cluster_partition_leader_id vectorized_cluster_partition_under_replicated_replicas Issue redpanda-data#15811. Issue redpanda-data/core-internal#677. (cherry picked from commit 27e6e2b)

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the tx metrics. This change enable aggregation on the partition label for almost all three metrics in this probe. Issue redpanda-data#15811. Issue redpanda-data/core-internal#677. (cherry picked from commit 65c0a89)

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the consensus object for two metrics: leader_for and reconfiguration changes in progress. This change enable aggregation on the partition label for these metrics if aggregation is turned on. Issue #15811. Issue redpanda-data/core-internal#677. (cherry picked from commit 04b1535)

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the partition probe object. This change enable aggregation on the partition label for almost all metrics in the partition problem, with only two excluded: vectorized_cluster_partition_leader_id vectorized_cluster_partition_under_replicated_replicas Issue #15811. Issue redpanda-data/core-internal#677. (cherry picked from commit 27e6e2b)

When metrics aggregation is turned on, we want to aggregate away the partition labels on most metrics: this wasn't occurring in the tx metrics. This change enable aggregation on the partition label for almost all three metrics in this probe. Issue #15811. Issue redpanda-data/core-internal#677. (cherry picked from commit 65c0a89)