Promagent on a standalone Tomcat shutdown issue

[WimLambrecht]

Hi,

I've visited several forums and could not find a similar issue (with javaagent in general that is).
I'm kinda stuck (in the last mile) for the moment ...

Some context and the issue:

I'm trying to use the 'promagent' instrumention in a standalone Tomcat context.
I've made an agent implementation based on your promagent-example. It's actually very similar to your 'ServletHook' -example. So it's affecting the Servlet and ServletFilter implementation.
I've provided this/my java agent on the Tomcat process (at startup) via the CATALINA_OPTS settings like this:

export CATALINA_OPTS=" $CATALINA_OPTS -javaagent:./promagent.jar=port=40111 "

Further:

• Running on a standalone Tomcat (8.5) on linux;
• The TOMCAT webapps folder contains an simple webapp (actually the 'sample.war' from Tomcat site examples). And it's the only webapps deployed (for now).
• The service is launched via 'catalina.sh start'.
• everything works fine; i can reach the 40111 metrics endpoint which provides sensible data.

The issue:

When execute a 'catalina.sh stop' to shutdown the app/service (again: with the CATALINA_OPS still in effect) ... :

• the main java process (the war gets undeployed) :
  02-Apr-2019 09:48:04.324 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
• the main webapp is not reacheable any longer, as expected BUT the main process (catalina) still keeps on running, and so is (or 'because of') the promagent. The latter is still reacheable and keeps on delivering metrics when asked for ...
• for clarity: i did not dive into the promagent-framework code (yet) - i wanted to ask first.
• maybe it's a deamon-thread or classloader-issue ... or (hopefully) some extra setting/OPTS that i've forgot ...

So:
the question is; how can a avoid this error and/or how to setup the promagent in a Tomcat standalone context.

P.S. & FYI:

• everything works fine when i run everything as a SpringBoot-ed app (with embedded Tomcat) as in your example. (Which is a different setup then my standalone Tomcat).
• BTW: using JAVA_OPTS instead of CATALINA_OPTS gives a 'Address already in use', and the main process + agent continuous. So, that's even worse. CATALINA_OPTS is not considered at 'catalina stop'-time.

Thank you for any pointers, help, ... !

Best regards,
Wim.

[fstab]

This sounds like a bug. It does not seem related to missing settings. The only thing that starts a thread is the built-in server https://github.com/fstab/promagent/blob/master/promagent-framework/promagent-internal/src/main/java/io/promagent/internal/BuiltInServer.java. If you start promagent without the port parameter this server will not be started. You could try that to check if this is the process preventing shutdown.

[WimLambrecht]

Indeed, after starting without the port, the process shuts down gracefully.
As a solution: maybe you can register a VM-shutdownhook that provides a httpServer.stop(...) to gracefully shutdown the httpServer instance?
P.S. not sure how this affects non-standalone-Tomcat usages of the promagent though (I did not look into it (yet) though).

[fstab]

It would be interesting to see if a shutdown hook works. If you try this and it works, please open a pull request.

Alternatively, there are other ways to expose the metrics (see https://github.com/fstab/promagent#exposing-metrics):

• You can deploy promagent.war and it will expose the metrics.
• Metrics are exposed via JMX, so you could for example use github.com/prometheus/jmx_exporter to expose the metrics.

[WimLambrecht]

A yeah, ok, that's right. I blindly went with the javaagent approach some time ago and forgot all about those alternatives. Since i already use the JMX exporter for some other cases - on the same Tomcat app and thus already present - i'll first go with that one and try that path.

The promagent.war approach like promising also, it fits the Catalina approach, and i'll definitely try that one also.

About the shutdown hook; if i find the time i'll definitely try that one too. And if it's valuable, i'll open a pull request.

Thanks for the help (and i'll confirm if those alternatives work for me) !