WildFly 10 example #87

cfrantsen · 2016-10-21T06:44:56Z

No description provided.

brian-brazil · 2016-10-21T07:54:40Z

example_configs/wildfly-10.yaml

+---
+lowercaseOutputName: true
+lowercaseOutputLabelNames: true
+whitelistObjectNames:


Given this is a framework, you shouldn't assume only the wildfly objects are of potential interest

So remove the whitelist or did I missunderstand you?

I'd remove it

brian-brazil · 2016-10-21T07:54:59Z

example_configs/wildfly-10.yaml

+    attrNameSnakeCase: true
+    name: wildfly_messaging_$2
+    labels:
+      name: $1


Call this queue or topic

brian-brazil · 2016-10-21T07:55:33Z

example_configs/wildfly-10.yaml

+    labels:
+      name: $1
+
+  - pattern: "^jboss.as<subsystem=transactions><>(number_of_.+):"


I'd not include the number_of_ in the output name. Every metric is the number of something.

cfrantsen · 2016-10-21T08:22:13Z

Updated with your suggestions, removing the whitelist increases scrape time with ~8x but since it's just an example config i guess that's not a real problem :)

brian-brazil · 2016-10-21T08:45:50Z

Thanks!

Starefossen · 2017-03-22T06:46:20Z

@cfrantsen I am trying to get this working with Wildfly 10 without any luck. I am wondering if you have a complete example you could share? The problem is the connection from jmx_exporter to the jmx management interface exposed by Wildlfy...

cfrantsen · 2017-03-22T07:22:25Z

I'm using the jmx_exporter as a java agent when running on WildFly. It requires some fiddling with start parameters to get working, below is an excerpt from the start script I'm using:

JBOSS_LOGMANAGER_JAR=$(echo $JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-*.jar)
JMX_EXPORTER_JAR=$(echo /opt/prometheus/jmx_exporter/jmx_prometheus_javaagent-*.jar)

if [ -f $JBOSS_LOGMANAGER_JAR -a -f $JMX_EXPORTER_JAR ]; then
    JBOSS_MODULES_SYSTEM_PKGS="$JBOSS_MODULES_SYSTEM_PKGS,org.jboss.logmanager"

    JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$JBOSS_LOGMANAGER_JAR"
    JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
    JAVA_OPTS="$JAVA_OPTS -javaagent:$JMX_EXPORTER_JAR=9102:/opt/prometheus/jmx_exporter/wildfly.yaml"
fi

Starefossen · 2017-03-22T08:42:11Z

Thanks Christian! Do you have to specify either hostPort or jmxUrl in the wildfly.yaml? I am using the wildfly-10.yaml from your pull request and I only see jvm_* and process_* metrics.

Here is a dump of my http://localhost:9102/metrics: gist.github.com/dbb983

cfrantsen · 2017-03-22T09:39:59Z

No that should not be needed when running it as a java agent. Have you tried a config file without explicit rules, that should give you a huge amount of metrics back.

Starefossen · 2017-03-22T10:01:58Z

I get much more output, but still nothing about Wildfly/Undertow: gist.github.com/9a8ff4.

How do you start your Wildfly? Are you using the standalone.sh command?

Starefossen · 2017-03-27T10:05:26Z

Sorry for bothering; I have made a minimal Wildfly jmx_exporter example using the official Wildfly Docker image and latest version of jmx_exporter which also does not show any Wildfly specific metrics.

From the Dockerfile source:

ADD jboss-logmanager-2.0.6.Final.jar /opt/jboss/wildfly
ADD jmx_prometheus_javaagent-0.9.jar /opt/jboss/wildfly
ADD prometheus-config.yaml /opt/jboss/wildfly

ENV JAVA_OPTS "-Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager \
               -Xbootclasspath/p:/opt/jboss/wildfly/jboss-logmanager-2.0.6.Final.jar \
               -javaagent:/opt/jboss/wildfly/jmx_prometheus_javaagent-0.9.jar=31500:/opt/jboss/wildfly/prometheus-config.yaml"

You can run this yourself using docker-compose up.

cfrantsen · 2017-03-28T07:03:13Z

We are running 0.7.3.298dcdc here right now so maybe it's a problem with recent versions? Perhaps try an older version and if that works create an issue.

Starefossen · 2017-03-28T07:50:12Z

Thanks a million @cfrantsen 💯 I only tried with v0.8 and v0.9, this is now working perfectly in v0.7!

cfrantsen · 2017-03-28T11:25:17Z

Could you perhaps create an issue regarding this as it can be fixed?

juliohm1978 · 2018-02-16T18:22:50Z

Anyone got this working for Wildfly 8.2.1? Using @cfrantsen's example it dies with the following message.

java.lang.IllegalArgumentException: Failed to load module "org.jboss.logmanager" for handler "FILE"
	at org.jboss.logmanager.config.AbstractPropertyConfiguration.<init>(AbstractPropertyConfiguration.java:67)
	at org.jboss.logmanager.config.HandlerConfigurationImpl.<init>(HandlerConfigurationImpl.java:54)
	at org.jboss.logmanager.config.LogContextConfigurationImpl.addHandlerConfiguration(LogContextConfigurationImpl.java:145)
	at org.jboss.logmanager.PropertyConfigurator.configureHandler(PropertyConfigurator.java:669)
	at org.jboss.logmanager.PropertyConfigurator.configureLogger(PropertyConfigurator.java:579)
	at org.jboss.logmanager.PropertyConfigurator.configure(PropertyConfigurator.java:517)
	at org.jboss.logmanager.PropertyConfigurator.configure(PropertyConfigurator.java:97)
	at org.jboss.logmanager.LogManager.readConfiguration(LogManager.java:300)
	at org.jboss.logmanager.LogManager.readConfiguration(LogManager.java:262)
	at java.util.logging.LogManager$3.run(LogManager.java:399)
	at java.util.logging.LogManager$3.run(LogManager.java:396)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396)
	at java.util.logging.LogManager.access$800(LogManager.java:145)
	at java.util.logging.LogManager$2.run(LogManager.java:345)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338)
	at java.util.logging.LogManager.getLogManager(LogManager.java:378)
	at java.util.logging.Logger.demandLogger(Logger.java:448)
	at java.util.logging.Logger.getLogger(Logger.java:502)
	at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.<clinit>(JmxCollector.java:38)
	at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.NoClassDefFoundError: org/jboss/modules/ModuleLoader
	at org.jboss.logmanager.config.AbstractPropertyConfiguration$ModuleFinder.getClassLoader(AbstractPropertyConfiguration.java:466)
	at org.jboss.logmanager.config.AbstractPropertyConfiguration.<init>(AbstractPropertyConfiguration.java:65)
	... 27 more

juliohm1978 · 2018-02-19T01:43:58Z

Finally got it working. If anyone else is stumbling on similar issues, remember: the following flags MUST ALL be included in JAVA_OPTS.

-Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager

-Xbootclasspath/p:/path/to/logmanager-X.X.X.Final.jar

-javaagent:/path/to/jmx_prometheus_javaagent.jar=NNNN:/path/to/config.yaml

Particularly, I was running in circles trying to define JBOSS_MODULES_SYSTEM_PKGS before starting wildfly. By the looks of standalone.conf, it should end up defining the -Djboss.modules.system.pkgs flag.

... Except, it isn't for some reason.

-Djboss.modules.system.pkgs=org.jboss.logmanager needs to be there. In order for this to work. Adding it to JAVA_OPTS myself resolved the issue.

I successfully tested it with Wildfly-8.2.1.Final and Wildfly-10.1.0.Final.

PS: If you have custom-handlers defined for logging, the above will also not work. Some of our installations need to define a custom logger to send logs to a remote graylog server via UDP. With a custom-handler defined, Wildfly's LogManager will not be able to bootstrap ahead of time for JMX Exporter.

brian-brazil reviewed Oct 21, 2016

View reviewed changes

Add WildFly 10 example

57bf061

brian-brazil merged commit 48088e9 into prometheus:master Oct 21, 2016

cfrantsen deleted the wildfly-example branch October 22, 2016 10:32

Starefossen mentioned this pull request Mar 28, 2017

JMX agent metrics from Wildfly/Undertow broken since v0.8 #128

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WildFly 10 example #87

WildFly 10 example #87

cfrantsen commented Oct 21, 2016

brian-brazil Oct 21, 2016

cfrantsen Oct 21, 2016

brian-brazil Oct 21, 2016

brian-brazil Oct 21, 2016

brian-brazil Oct 21, 2016

cfrantsen commented Oct 21, 2016

brian-brazil commented Oct 21, 2016

Starefossen commented Mar 22, 2017

cfrantsen commented Mar 22, 2017

Starefossen commented Mar 22, 2017 •

edited

cfrantsen commented Mar 22, 2017

Starefossen commented Mar 22, 2017 •

edited

Starefossen commented Mar 27, 2017 •

edited

cfrantsen commented Mar 28, 2017

Starefossen commented Mar 28, 2017

cfrantsen commented Mar 28, 2017

juliohm1978 commented Feb 16, 2018 •

edited

juliohm1978 commented Feb 19, 2018 •

edited

WildFly 10 example #87

WildFly 10 example #87

Conversation

cfrantsen commented Oct 21, 2016

brian-brazil Oct 21, 2016

Choose a reason for hiding this comment

cfrantsen Oct 21, 2016

Choose a reason for hiding this comment

brian-brazil Oct 21, 2016

Choose a reason for hiding this comment

brian-brazil Oct 21, 2016

Choose a reason for hiding this comment

brian-brazil Oct 21, 2016

Choose a reason for hiding this comment

cfrantsen commented Oct 21, 2016

brian-brazil commented Oct 21, 2016

Starefossen commented Mar 22, 2017

cfrantsen commented Mar 22, 2017

Starefossen commented Mar 22, 2017 • edited

cfrantsen commented Mar 22, 2017

Starefossen commented Mar 22, 2017 • edited

Starefossen commented Mar 27, 2017 • edited

cfrantsen commented Mar 28, 2017

Starefossen commented Mar 28, 2017

cfrantsen commented Mar 28, 2017

juliohm1978 commented Feb 16, 2018 • edited

juliohm1978 commented Feb 19, 2018 • edited

Starefossen commented Mar 22, 2017 •

edited

Starefossen commented Mar 22, 2017 •

edited

Starefossen commented Mar 27, 2017 •

edited

juliohm1978 commented Feb 16, 2018 •

edited

juliohm1978 commented Feb 19, 2018 •

edited