[improve] [pip] PIP-355: Enhancing Broker-Level Metrics for Pulsar (a…

…pache#22778)

Co-authored-by: Dragos Misca <dragosvictor@users.noreply.github.com>

pip/pip-355.md

@@ -0,0 +1,36 @@
+# PIP-355: Enhancing Broker-Level Metrics for Pulsar
+
+# Background Knowledge
+Pulsar provides broker-level, namespace-level, and topic-level metrics to monitor and analyze the behavior of the Pulsar service. These metrics are accessible through the Prometheus metrics endpoint. Detailed explanations of all metrics can be found on the Pulsar website: [Pulsar Metrics Reference](https://pulsar.apache.org/docs/3.2.x/reference-metrics/)
+
+# Motivation
+Within Pulsar's current metrics framework, the `pulsar_out_bytes_total` metric is utilized to expose the total bytes dispatched by the broker to consumers. However, there are notable limitations and challenges associated with this metric:
+- Inclusion of system subscriptions in the total bytes out, alongside user subscriptions, complicates accurate calculation of user-specific data.
+- The granularity of the metric (namespace-level vs. topic-subscription level) impacts the scalability and resource consumption when calculating cluster-level total out bytes.
+
+# Goals
+This proposal aims to address the following objectives:
+- Simplify the process of calculating cluster-level total out bytes.
+- Enable the calculation of total out bytes dispatched to system subscriptions.
+
+# High-Level Design
+To achieve the outlined goals, the proposal introduces two new broker-level metrics:
+- `pulsar_broker_out_bytes_total{system_subscription="true|false"}`: Represents the total out bytes dispatched by the broker to consumers. The label `system_subscription="false"` represents total traffic dispatched to user subscriptions, while `system_subscription="true"` represents total traffic dispatched to system cursors and cursor names added by `additionalSystemCursorNames` introduced in [PIP-349](https://github.com/apache/pulsar/pull/22651).
+- `pulsar_broker_in_bytes_total{system_topic="true|false"}`: Tracks the total in bytes sent by producers to the broker. The label `system_topic="false"` represents total traffic from user topics, while `system_topic="true"` represents total traffic from system topics.
+
+# Detailed Design
+The implementation involves the introduction of the following broker-level metrics:
+- `pulsar_broker_out_bytes_total{system_subscription="true|false"}`: Aggregates the total out bytes from all topics, presented as a broker-level metric. 
+- `pulsar_broker_in_bytes_total{system_topic="true|false"}`: Calculation of total in bytes across all topics.
+
+# Metrics
+The proposal includes the addition of two new broker-level metrics:
+- `pulsar_broker_out_bytes_total{system_subscription="true|false"}`
+- `pulsar_broker_in_bytes_total{system_topic="true|false"}`
+
+# Backward & Forward Compatibility
+The proposed changes ensure full compatibility with existing systems and pave the way for seamless integration with future enhancements.
+
+# Links
+- Mailing List discussion thread: https://lists.apache.org/thread/n3vvh6pso9ml7sg3qpww870om5vcfnpv
+- Mailing List voting thread: https://lists.apache.org/thread/h4rjcv77wppz96gc31cpr3hw17v9jc4o