broker error for prometheus metrics #3112
Labels
type/bug
The PR fixed a bug or issue reported a bug
Comments
ivankelly
added
type/bug
|
We have something similar:
When doing a port forward on my pod: I can see on http://localhost:8080/metrics/. Some of the metrics like |
|
Same story. I have an error in Prometheus Any suggestions? |
|
We had to update prometheus and grafana, and refactor the existing configuration for both until it was compatible with the last version in order to get ride of these errors. |
jiazhai
pushed a commit
that referenced
this issue
Aug 26, 2019
… fixed format issue in metricWithRemoteCluster; added test for Prometheus types (#4183) Fixes #3112 ### Motivation In this pull request, I set out to fix #3112, which is caused by duplicate TYPE statements in the metrics output which leads to parsing of Prometheus metrics to fail in recent versions of Prometheus. Because of this, Prometheus will report the broker target as down. Since I started looking at this, the type definitions have been removed (#4136) from topics metric output. I think these types are useful in Prometheus and have added them back in. While testing this fix in my geo-replicated setup, I found a format in error (missing quote and comma) in the TopicStats.metricWithRemoteCluster method. This pull request includes a fix for that issue. I have also added a new test to PrometheusMetricsTest.java that fails without these changes but passed with them. ### Modifications I added a static HashMap to TopicStats to keep track of the TYPEs that have been output. All writing of the TYPE for topics and namespaces is done with the TopicStats.metricType method. I modified that method to update the HashMap and only print the TYPE out for the first occurrence of the metric name. I also added a method reset the HashMap, which gets called in NamespaceStatsAggregator.generate. ### Verifying this change This change added tests and can be verified as follows: - Added testDuplicateMetricTypeDefinitions which checks for: - duplicate TYPE definitions in the Prometheus output - validates that no TYPE definition appears after the first metric sample - ensures that all metrics have a defined type I execute the test twice to make sure the resetting of the HashMap of the already seen metric type definitions works correctly. This test passes for me reliably (both occurrences). I have confirmed using promtool that the metrics output will now parse without error using versions 2.7.1 and 2.9.2 (which is the latest). There are many warnings around missing HELP definitions and metrics using reserved suffixes (ex. _count), but no errors. In addition, I have patched 2.3.1 with this fix and am currently running it in my cluster. Prometheus (2.7.1) successfully parses the metrics and I am able to see namespace and topic-level metrics.
jiazhai
pushed a commit
that referenced
this issue
Aug 28, 2019
… fixed format issue in metricWithRemoteCluster; added test for Prometheus types (#4183) Fixes #3112 In this pull request, I set out to fix #3112, which is caused by duplicate TYPE statements in the metrics output which leads to parsing of Prometheus metrics to fail in recent versions of Prometheus. Because of this, Prometheus will report the broker target as down. Since I started looking at this, the type definitions have been removed (#4136) from topics metric output. I think these types are useful in Prometheus and have added them back in. While testing this fix in my geo-replicated setup, I found a format in error (missing quote and comma) in the TopicStats.metricWithRemoteCluster method. This pull request includes a fix for that issue. I have also added a new test to PrometheusMetricsTest.java that fails without these changes but passed with them. I added a static HashMap to TopicStats to keep track of the TYPEs that have been output. All writing of the TYPE for topics and namespaces is done with the TopicStats.metricType method. I modified that method to update the HashMap and only print the TYPE out for the first occurrence of the metric name. I also added a method reset the HashMap, which gets called in NamespaceStatsAggregator.generate. This change added tests and can be verified as follows: - Added testDuplicateMetricTypeDefinitions which checks for: - duplicate TYPE definitions in the Prometheus output - validates that no TYPE definition appears after the first metric sample - ensures that all metrics have a defined type I execute the test twice to make sure the resetting of the HashMap of the already seen metric type definitions works correctly. This test passes for me reliably (both occurrences). I have confirmed using promtool that the metrics output will now parse without error using versions 2.7.1 and 2.9.2 (which is the latest). There are many warnings around missing HELP definitions and metrics using reserved suffixes (ex. _count), but no errors. In addition, I have patched 2.3.1 with this fix and am currently running it in my cluster. Prometheus (2.7.1) successfully parses the metrics and I am able to see namespace and topic-level metrics. (cherry picked from commit 8d32b58)
|
Seems the problem can still happens, but i can't get the restful response, broker is crashed at that time. |
|
I think this issue has been fixed in the latest Pulsar release. Close this issue now. Please reopen one if there are still issues. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected behavior
I deployed pulsar with helm and minikube configuration in kubernetes. I tried multiple times, I having problem with one of the brokers, sometimes both brokers. The problem has been consistent for three deployments so far.
Steps to reproduce
deploy pulsar with helm
System configuration
Pulsar version: x.y
image: apachepulsar/pulsar-all
Error in Broker.log
06:40:34.280 [pulsar-web-30-30] INFO org.eclipse.jetty.server.RequestLog - 10.244.6.29 - - [03/Dec/2018:06:40:34 +0000] "GET //xx.xx.xx.xx:8080/metrics HTTP/1.1" 302 0 "-" "Prometheus/1.6.3" 1
06:40:34.287 [prometheus-stats-31-1] ERROR org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet - Failed to generate prometheus stats
org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:197) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:420) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:375) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.SelectChannelEndPoint$3.run(SelectChannelEndPoint.java:107) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:193) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:283) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) ~[org.eclipse.jetty-jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) ~[org.eclipse.jetty-jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) ~[org.eclipse.jetty-jetty-util-9.3.11.v20160721.jar:9.3.11.v20160721]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [org.apache.pulsar-pulsar-functions-metrics-2.2.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_171]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_171]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_171]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.8.0_171]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_171]
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:175) ~[org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
... 13 more
Error in Prometheus
text format parsing error in line 182: second TYPE line for metric name "pulsar_subscription_back_log", or TYPE reported after samples
The text was updated successfully, but these errors were encountered: