[HUDI-210] hudi-support-prometheus-pushgateway

[liujinhui1994]

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

hudi support prometheus/pushgateway

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[liujinhui1994]

[liujinhui1994]

[liujinhui1994]

@leesf Please review

[leesf]

rerun tests

[liujinhui1994]

rerun tests

[leesf]

LGTM, merging.

[sbernauer]

Hi, thanks for this PR!
I added simpleclient-0.8.0.jar simpleclient_dropwizard-0.8.0.jar simpleclient_httpserver-0.8.0.jar and simpleclient_pushgateway-0.8.0.jar to the classpath. If i try to use the prometheus exporter i get the following error. Do you know why this happens?

Caused by: java.lang.NoSuchMethodError: io.prometheus.client.dropwizard.DropwizardExports.<init>(Lorg/apache/hudi/com/codahale/metrics/MetricRegistry;)V
        at org.apache.hudi.metrics.prometheus.PrometheusReporter.<init>(PrometheusReporter.java:49)
        at org.apache.hudi.metrics.MetricsReporterFactory.createReporter(MetricsReporterFactory.java:75)
        at org.apache.hudi.metrics.Metrics.<init>(Metrics.java:45)
        at org.apache.hudi.metrics.Metrics.init(Metrics.java:73)
        at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics.<init>(HoodieDeltaStreamerMetrics.java:44)
        at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:227)
        at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:579)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[liujinhui1994]

Are you using deltastreamer?
Use to recompile and try again

[sbernauer]

Yes, im using deltastreamer, freshly compiled from source of the master branch.
The new config:

    hoodie.metrics.on=true
    hoodie.metrics.reporter.type=PROMETHEUS

[liujinhui1994]

Please use 'include'
and try to recompile

[sbernauer]

I'm sorry, i dont get it. I use mvn clean package -DskipTests -DskipITs -Dscala-2.12 for compilation. What do you mean by

Please use 'include'

Edit: Oh you mean adding the scope include in the pom for the 4 dependencies?

[liujinhui1994]

yes

[sbernauer]

My setup has no access to the internet, so i downloaded the libs from mvn central and included them in my docker images. Is the version 0.8.0 correct?

$ ls simpleclient*
simpleclient-0.8.0.jar  simpleclient_common-0.8.0.jar  simpleclient_dropwizard-0.8.0.jar  simpleclient_httpserver-0.8.0.jar  simpleclient_pushgateway-0.8.0.jar

[liujinhui1994]

0.8.0 correct

[sbernauer]

The constructor of DropwizardExports seems to be stable and I'm wondering what is going wrong here (It does not seem like a version mismatch to me). https://github.com/prometheus/client_java/blob/570f8bb70ae6f5a0a950dc563b68fb495d2f4d69/simpleclient_dropwizard/src/main/java/io/prometheus/client/dropwizard/DropwizardExports.java#L30
Mabye org/apache/hudi/com/codahale/metrics/MetricRegistry (see exception) instead of com/codahale/metrics/MetricRegistry is a problem?

[sbernauer]

If i uncompress my simpleclient_dropwizard-0.8.0.jar and look inside the class all seems correct:

$ javap DropwizardExports.class 
Compiled from "DropwizardExports.java"
public class io.prometheus.client.dropwizard.DropwizardExports extends io.prometheus.client.Collector implements io.prometheus.client.Collector$Describable {
  public io.prometheus.client.dropwizard.DropwizardExports(com.codahale.metrics.MetricRegistry);
  public io.prometheus.client.dropwizard.DropwizardExports(com.codahale.metrics.MetricRegistry, io.prometheus.client.dropwizard.samplebuilder.SampleBuilder);
[...]
}

[liujinhui1994]

If you use the include tag to include the required dependencies, it should be correct. Let me see why

[sbernauer]

I don't use the include option but instead guarantee that all drivers and executors have the libs and correctly use them via SPARK_DIST_CLASSPATH.

[liujinhui1994]

It’s because I missed to configure less dependencies, it will be fixed soon
@sbernauer

[sbernauer]

Thanks a lot @UZi5136225 for your fast response!

[liujinhui1994]

#1942 @sbernauer You can try this PR

[sbernauer]

Thanks @UZi5136225, this fixed the problem!
I dont understand why, because i added all the libs below in the correct version to the classpath. But it works :)

mvn dependency:tree | grep simpleclient
[INFO] +- io.prometheus:simpleclient:jar:0.8.0:compile
[INFO] +- io.prometheus:simpleclient_httpserver:jar:0.8.0:compile
[INFO] |  \- io.prometheus:simpleclient_common:jar:0.8.0:compile
[INFO] +- io.prometheus:simpleclient_dropwizard:jar:0.8.0:compile
[INFO] +- io.prometheus:simpleclient_pushgateway:jar:0.8.0:compile

[sbernauer]

@UZi5136225 i just noticed that i get a java.lang.ClassNotFoundException: io.prometheus.client.exporter.common.TextFormat when trying to access the interface. I think we should also add (i will give it a try)

<include>io.prometheus:simpleclient_common</include>

[liujinhui1994]

Yes, this configuration needs to be added

[sbernauer]

Hi @UZi5136225 it is currently not possible to set the metric with hoodie.metrics.prometheus.metric.prefix. Is this intended?

[leesf]

Hi @UZi5136225 it is currently not possible to set the metric with hoodie.metrics.prometheus.metric.prefix. Is this intended?

@UZi5136225 would you please take a look at this?

[liujinhui1994]

Right Prometheus does not support, these are specially made for pushgateway
@sbernauer