Conductor server metrics not getting published

[clari-akhilesh]

We're looking to monitor the performance of conductor servers and workers in our production environment. The documentation lists some of the metrics that are published from the servers and workers. While the metrics for the workers look good, I don't see any metrics for conductor servers.
Digging through the code, I see that you use Netflix spectator library to record the metrics and the registry is being created as follows:
private static Registry registry = Spectator.globalRegistry();
The javadoc for this method is as follows:
Returns the global composite registry. This method can be used for use-cases where it is necessary to get a static reference to a registry. It will not do anything unless other registries are added.
It looks like we're not adding any registries and I suspect that is the reason we're not seeing any metrics on the server side. Here are some of the metrics I expect to see on the server side - https://netflix.github.io/conductor/metrics/
I hope this is addressed in the next release as we don't have any visibility on how the conductor servers are performing. Thank you in advance!

[clari-akhilesh]

Any update on this ticket? It would be really helpful to have the server metrics pushed to JMX

[mudit3774]

@clari-akhilesh did you manage to figure this out ? @cyzhao @v1r3n are we missing something here ? We also were able to export worker metrics by adding Metrics3 registry but how should we approach the same on the server ?

[cyzhao]

Both Monitors and WorkflowTaskMetrics use Spectator.globalRegistry(), please make sure that you have a spectator registry dependency added to server deployment, check out: http://netflix.github.io/spectator/en/latest/registry/metrics3/

[JasperB]

@cyzhao We also have the same requirement, we want to check the metrics for server and worker, is there any detail guide to see the metrics? it's really helpful for monitoring to see how't the server and worker performing. Looking forward to getting reply.

[clari-akhilesh]

@cyzhao , what do you mean by make sure that you have a spectator registry dependency added to server deployment? Where should this be done? We currently deploy both the server and workers on Jetty.

[cyzhao]

@clari-akhilesh as described in http://netflix.github.io/spectator/en/latest/registry/metrics3/, add com.netflix.spectator:spectator-reg-metrics3:0.69.0 to gradle and provide MetricsRegistry as Registry
in your Guice module.

[clari-akhilesh]

we currently use maven war artifact in our deployment. if I understand correctly, you don't have any bindings for the Registry class in the code. are you suggesting that I submit a PR with the required metrics3 bindings? I'm happy to do so, if that's what you're suggesting. Just want to make sure I understand you correctly. Thanks!

[rileymichael]

@clari-akhilesh did you end up finding a way to provide a Spectator registry while still using the prebuilt jar?

[clari-akhilesh]

@rileymichael , unfortunately not. waiting to hear from @cyzhao if a code change is needed to accomplish this.

[cyzhao]

@clari-akhilesh sorry for the late response, I don't think you need to submit a PR for this, you can initialize the Registry and add it to the global registry via ((CompositeRegistry)Spectator.globalRegistry()).add(...), sample code for creating metrics3 registry: https://github.com/brharrington/spectator-examples/blob/master/metrics3/src/main/java/com/netflix/spectator/example/metrics3/Main.java

[mohelsaka]

@cyzhao
Does this mean we have to change the code (possibly bootstrap.Main.java file) in order to add registries to Spectator.globalRegistry.

Is there a better way to do this without chaging the code?

[mohelsaka]

@clari-akhilesh
Here's how I figured it out. In case you're still interested

Conductor Server has a feature to load additional modules dynamically, this feature can be controlled using this the following configuration.

# Additional modules (optional)
# conductor.additional.modules=class_extending_com.google.inject.AbstractModule

Using this feature, you can basically create your own AbstractModule that overides configure function and registers whatever Spectator metrics registery as @cyzhao has mentioned above:
Spectator.globalRegistry().add(...).

Since you mentioned that you're using maven, then you can create an independent maven project that produces a JAR file with all the dependecies using maven-shade-plugin. And then you'll need to change the the way conductor server runs in this file here.
Instead of:
java -jar conductor-server-*-all.jar $config_file
Use:
java -cp conductor-server-2.6.0-all.jar:metrics-initializer-lib-1.0.0.jar com.netflix.conductor.bootstrap.Main $config_file

It's a bit complecated, but it's doable. I used this way to publish promethues metrics from conductor, with very little changes to conductor server, I can share how I manged to do it in case you're interested.

[rileymichael]

@mohelsaka Thanks for the explanation! I'd be very interested in seeing how you are publishing metrics w/ prometheus.

[cyzhao]

@kishorebanala @apanicker-nflx please make sure this is covered in the new documentation we are working on.

[clari-akhilesh]

@mohelsaka , thanks for the explanation. I'll be sure to give this a try.

[apanicker-nflx]

Please refer the documentation here to publish metrics from conductor.

[jdoupe]

Not quite up to spec with the docs, but workable solution for providing prometheus endpoint:
https://github.com/Netflix/conductor/compare/master...jdoupe:add-prometheus-support?expand=1

[pchowdad]

Not quite up to spec with the docs, but workable solution for providing prometheus endpoint:
https://github.com/Netflix/conductor/compare/master...jdoupe:add-prometheus-support?expand=1

Thank you for sharing. Wondering why this is not merged to master yet. Do we have any other approach?

[marcocrasso]

For those interested in this support working with Prometheus:

Using latest main branch, and after consulting:

1. https://micrometer.io/docs/registry/prometheus#_installing (just need Section 1 and 2 sentences of Section 2)
2. https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html#production-ready-endpoints-exposing-endpoints

DEV

in server/build.gradle file add:

implementation 'org.springframework.boot:spring-boot-starter-actuator'

in server/src/main/resources/application.properties add:

conductor.metrics-prometheus.enabled=true
management.endpoints.web.exposure.include=prometheus,health,info,metrics

** EDITED **
in contribs/metrics/PrometheusMetricsConfiguration.java

Remove the instantiation of Prometheus registry, and let Spring inject the actuator one:

public PrometheusMetricsConfiguration(MeterRegistry meterRegistry) {
    LOGGER.info("Prometheus metrics module initialized");

    final MicrometerRegistry metricsRegistry = new MicrometerRegistry(meterRegistry);

then hit at root:

./gradlew server
visit: http://localhost:8080/actuator and follow the links.

PRD

For production, build a docker image with the change in server/build.gradle and then configure your container as showed.

[FingerLiu]

I solved this by add the following packages to pom.xml since I am using maven.

        <dependency>
            <groupId>com.netflix.spectator</groupId>
            <artifactId>spectator-reg-metrics3</artifactId>
            <version>0.90.0</version>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
            <version>2.5.0</version>
        </dependency>
        <dependency>
            <groupId>com.netflix.spectator</groupId>
            <artifactId>spectator-reg-micrometer</artifactId>
            <version>0.130.0</version>
        </dependency>