-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Issue 3112][Broker Prometheus Metrics]Fix Duplicate Prometheus TYPE fix and add test #4183
Conversation
@cdbartholomew I think this PR #4136 fixes the issue already |
@jerrypeng that PR (#4136) removes the TYPEs altogether, so now there are no type definitions for namespace and topic metrics. This PR restores the TYPEs and makes sure they aren't duplicated. It also includes a test. I see that the PR has failed the unit tests. I will take a look at that. |
… fixed format issue in metricWithRemoteCluster; added test for Prometheus types
1e8fe64
to
7edf929
Compare
@sijie I have rebased this PR as you requested. |
// PulsarFunctionsProcessTest.pulsar-test-suite run integration tests // org.apache.pulsar.client.api.DispatcherBlockConsumerTest.testBrokerSubscriptionRecovery run java8 tests |
@jerrypeng @merlimat can you guys please take a look at this pull request? |
ping @merlimat @jerrypeng |
For the time of 2.4.1 release, I would like to merge this fix. |
… fixed format issue in metricWithRemoteCluster; added test for Prometheus types (#4183) Fixes #3112 In this pull request, I set out to fix #3112, which is caused by duplicate TYPE statements in the metrics output which leads to parsing of Prometheus metrics to fail in recent versions of Prometheus. Because of this, Prometheus will report the broker target as down. Since I started looking at this, the type definitions have been removed (#4136) from topics metric output. I think these types are useful in Prometheus and have added them back in. While testing this fix in my geo-replicated setup, I found a format in error (missing quote and comma) in the TopicStats.metricWithRemoteCluster method. This pull request includes a fix for that issue. I have also added a new test to PrometheusMetricsTest.java that fails without these changes but passed with them. I added a static HashMap to TopicStats to keep track of the TYPEs that have been output. All writing of the TYPE for topics and namespaces is done with the TopicStats.metricType method. I modified that method to update the HashMap and only print the TYPE out for the first occurrence of the metric name. I also added a method reset the HashMap, which gets called in NamespaceStatsAggregator.generate. This change added tests and can be verified as follows: - Added testDuplicateMetricTypeDefinitions which checks for: - duplicate TYPE definitions in the Prometheus output - validates that no TYPE definition appears after the first metric sample - ensures that all metrics have a defined type I execute the test twice to make sure the resetting of the HashMap of the already seen metric type definitions works correctly. This test passes for me reliably (both occurrences). I have confirmed using promtool that the metrics output will now parse without error using versions 2.7.1 and 2.9.2 (which is the latest). There are many warnings around missing HELP definitions and metrics using reserved suffixes (ex. _count), but no errors. In addition, I have patched 2.3.1 with this fix and am currently running it in my cluster. Prometheus (2.7.1) successfully parses the metrics and I am able to see namespace and topic-level metrics. (cherry picked from commit 8d32b58)
Fixes #3112
Motivation
In this pull request, I set out to fix #3112, which is caused by duplicate TYPE statements in the metrics output which leads to parsing of Prometheus metrics to fail in recent versions of Prometheus. Because of this, Prometheus will report the broker target as down.
Since I started looking at this, the type definitions have been removed (#4136) from topics metric output. I think these types are useful in Prometheus and have added them back in.
While testing this fix in my geo-replicated setup, I found a format in error (missing quote and comma) in the TopicStats.metricWithRemoteCluster method. This pull request includes a fix for that issue.
I have also added a new test to PrometheusMetricsTest.java that fails without these changes but passed with them.
Modifications
I added a static HashMap to TopicStats to keep track of the TYPEs that have been output. All writing of the TYPE for topics and namespaces is done with the TopicStats.metricType method. I modified that method to update the HashMap and only print the TYPE out for the first occurrence of the metric name. I also added a method reset the HashMap, which gets called in NamespaceStatsAggregator.generate.
Verifying this change
This change added tests and can be verified as follows:
I execute the test twice to make sure the resetting of the HashMap of the already seen metric type definitions works correctly. This test passes for me reliably (both occurrences).
I have confirmed using promtool that the metrics output will now parse without error using versions 2.7.1 and 2.9.2 (which is the latest). There are many warnings around missing HELP definitions and metrics using reserved suffixes (ex. _count), but no errors.
In addition, I have patched 2.3.1 with this fix and am currently running it in my cluster. Prometheus (2.7.1) successfully parses the metrics and I am able to see namespace and topic-level metrics.
Does this pull request potentially affect one of the following parts:
If
yes
was chosen, please highlight the changesDocumentation