[SUPPORT] Metrics Prometheus pushgateway

[PavelPetukhov]

I have discovered that you've added prometheus related changes
like here https://issues.apache.org/jira/browse/HUDI-210

But unfortunately there is no documentation related to pushing hudi metrics to Prometheus Push Gateway
https://hudi.apache.org/docs/metrics.html#hoodiemetrics

What parameters should be set in order to do that?

Environment Description

• Hudi version : 0.6.0

• Spark version : 2.4.7

• Hadoop version : 2.7

• Storage (HDFS/S3/GCS..) : hdfs

• Running on Docker? (yes/no) : no

[bvaradar]

Here is an example config :

--hoodie-conf hoodie.metrics.on=true --hoodie-conf hoodie.metrics.reporter.type=PROMETHEUS_PUSHGATEWAY --hoodie-conf hoodie.metrics.pushgateway.host=<PUSH-GATEWAY-HOST> --hoodie-conf hoodie.metrics.pushgateway.port=9091 --hoodie-conf hoodie.metrics.pushgateway.job.name=<JOB_NAME> --hoodie-conf hoodie.metrics.pushgateway.random.job.name.suffix=false --hoodie-conf hoodie.metrics.pushgateway.delete.on.shutdown=false

[PavelPetukhov]

Thanks a lot for the reply

I my case I am getting the following error:

1/04/26 13:29:52 WARN prometheus.PushGatewayReporter: Can't push monitoring information to pushGateway
java.net.UnknownHostException: https
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
at io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:305)
at io.prometheus.client.exporter.PushGateway.pushAdd(PushGateway.java:160)
at org.apache.hudi.metrics.prometheus.PushGatewayReporter.report(PushGatewayReporter.java:72)
at org.apache.hudi.metrics.prometheus.PushGatewayMetricsReporter.report(PushGatewayMetricsReporter.java:64)
at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:64)
at org.apache.hudi.metrics.Metrics.lambda$new$0(Metrics.java:57)
at java.lang.Thread.run(Thread.java:748)

my parameters are
--hoodie-conf hoodie.metrics.on=true
--hoodie-conf hoodie.metrics.reporter.type=PROMETHEUS_PUSHGATEWAY
--hoodie-conf hoodie.metrics.pushgateway.host=https://pushgateway_url.com/
--hoodie-conf hoodie.metrics.pushgateway.port=8080
--hoodie-conf hoodie.metrics.pushgateway.job.name=test_job_1
--hoodie-conf hoodie.metrics.pushgateway.random.job.name.suffix=false
--hoodie-conf hoodie.metrics.pushgateway.delete.on.shutdown=false

[nsivabalan]

@bvaradar : while you respond to author, can you add a faq entry(or some other documentation) on how to push hudi metrics to Prometheus Push Gateway.

[PavelPetukhov]

@bvaradar

Any ideas why I am getting
PushGatewayReporter: Can't push monitoring information to pushGateway

?

[n3nash]

@bvaradar Could you look into this please ?

[n3nash]

@PavelPetukhov This looks like the push gateway URL has not been configured correctly. The hoodie-conf hoodie.metrics.pushgateway.host probably expects a hostname instead of a http URL. Can you see if this is the issue ?

[vinothchandar]

Closing due to inactivity