Glowroot exceptions accessing PlatformMBeanServer with WebSphere9

[FJGL]

Environment: WebSphere9.0.0.5 64 bits Windows7 and IBM JDK8
Lots of exceptions because the ManagementFactory.class of this JVM has no longer a private Field for the PlatformMBeanServer. Instead, it is inside a nested static class ServerHolder.

2018-02-27 09:38:19.648 INFO org.glowroot - Glowroot version: 0.10.3, built 2018-02-23 05:08:03 +0000
2018-02-27 09:38:19.652 INFO org.glowroot - Java version: 1.8.0_144
2018-02-27 09:38:21.948 WARN org.glowroot.agent.init.AgentModule - one or more java.lang.Runnable or java.util.concurrent.Callable implementations were loaded before Glowroot instrumentation could be applied to them: java.util.concurrent.ThreadPoolExecutor$Worker, java.util.concurrent.Executors$RunnableAdapter, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask, java.util.concurrent.FutureTask. This may prevent Glowroot from capturing async requests that span multiple threads.
2018-02-27 09:38:21.949 WARN org.glowroot.agent.init.AgentModule - org.glowroot.agent.jul.Logger was loaded before Glowroot instrumentation could be applied to it. This may prevent Glowroot from capturing JUL logging.
2018-02-27 09:38:22.194 WARN o.glowroot.agent.init.GaugeCollector - error accessing mbean attribute: java.lang:type=Memory HeapMemoryUsage.used
java.lang.NoSuchFieldException: platformServer
at java.lang.Class.getDeclaredFieldImpl(Native Method)
at java.lang.Class.getDeclaredField(Class.java:754)
at org.glowroot.agent.util.LazyPlatformMBeanServer.waitForContainerToCreatePlatformMBeanServer(LazyPlatformMBeanServer.java:217)
at org.glowroot.agent.util.LazyPlatformMBeanServer.ensureInit(LazyPlatformMBeanServer.java:194)
at org.glowroot.agent.util.LazyPlatformMBeanServer.getAttribute(LazyPlatformMBeanServer.java:136)
at org.glowroot.agent.init.GaugeCollector.collectGaugeValues(GaugeCollector.java:194)
at org.glowroot.agent.init.GaugeCollector.collectGaugeValues(GaugeCollector.java:164)
at org.glowroot.agent.init.GaugeCollector.runInternal(GaugeCollector.java:121)
at org.glowroot.agent.shaded.org.glowroot.common.util.ScheduledRunnable.run(ScheduledRunnable.java:49)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:319)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:191)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:795)

Maybe this quick hack can solve it (I can't build the git project at this moment in my environment, sorry)

private static void waitForContainerToCreatePlatformMBeanServer(Stopwatch stopwatch)
        throws Exception {
    String platformMBeanServerFieldName =
            AppServerDetection.isIbmJvm() ? "platformServer" : "platformMBeanServer";
    Field platformMBeanServerField = null;
    try {
    	platformMBeanServerField=ManagementFactory.class.getDeclaredField(platformMBeanServerFieldName);
    } catch (NoSuchFieldException nfe) {
    	logger.warn("No field "+platformMBeanServerFieldName+" present in ManagementFactory" );
        if (AppServerDetection.isIbmJvm()) {
			platformMBeanServerField = Class.forName("java.lang.management.ManagementFactory$ServerHolder").getClass()
						.getDeclaredField(platformMBeanServerFieldName);
		}
    }
    platformMBeanServerField.setAccessible(true);
    stopwatch.start();
    while (stopwatch.elapsed(SECONDS) < 60) {
        if (platformMBeanServerField.get(null) != null) {
			logger.warn("*********** ManagementFactory$ServerHolder found with " + platformMBeanServerField.get(null));
            return;
        }
        Thread.sleep(100);
    }
    logger.error("platform mbean server was never created by container");
}

Regards

[trask]

Hi @FJGL, thanks for reporting this! I think it should be fixed now in the latest snapshot release glowroot-agent-0.10.4-SNAPSHOT-dist.zip.

[trask]

I posted too soon, the building was still running, but it is available now glowroot-agent-0.10.4-SNAPSHOT-dist.zip.

[FJGL]

Yes, I saw it and I was waiting for the new building. Thanks

[FJGL]

Hi, It's working now. THANKS!!
Only this error on startup:

2018-02-28 10:03:40.832 INFO org.glowroot - Glowroot version: 0.10.4-SNAPSHOT, commit 5490ea1, built 2018-02-28 08:24:53 +0000
2018-02-28 10:03:40.834 INFO org.glowroot - Java version: 1.8.0_144
2018-02-28 10:03:42.949 WARN org.glowroot.agent.init.AgentModule - java.lang.management.ManagementFactory was loaded before Glowroot instrumentation could be applied to it. This will likely prevent Glowroot from functioning properly.
2018-02-28 10:03:42.972 WARN org.glowroot.agent.init.AgentModule - one or more java.lang.Runnable or java.util.concurrent.Callable implementations were loaded before Glowroot instrumentation could be applied to them: java.util.concurrent.ThreadPoolExecutor$Worker, java.util.concurrent.Executors$RunnableAdapter, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask, java.util.concurrent.FutureTask. This may prevent Glowroot from capturing async requests that span multiple threads.
2018-02-28 10:03:42.973 WARN org.glowroot.agent.init.AgentModule - org.glowroot.agent.jul.Logger was loaded before Glowroot instrumentation could be applied to it. This may prevent Glowroot from capturing JUL logging.
2018-02-28 10:03:43.026 INFO org.glowroot - plugins loaded: Cassandra Plugin, Elasticsearch Plugin, Executor Plugin, Grails Plugin, Hibernate Plugin, HTTP Client Plugin, Java HTTP Server Plugin, JAX-RS Plugin, Jdbc Plugin, JMS Plugin, JSF Plugin, JSP Plugin, Logger Plugin, Play Plugin, Netty Plugin, Quartz Plugin, Redis Plugin, Servlet Plugin, Spring Plugin, Struts Plugin
2018-02-28 10:03:44.796 INFO org.glowroot - creating glowroot schema ...
2018-02-28 10:03:45.520 INFO org.glowroot - glowroot schema created

2018-02-28 10:04:43.615 ERROR o.g.a.util.LazyPlatformMBeanServer - platform mbean server was never created by container

2018-02-28 10:04:46.741 INFO org.glowroot - UI listening on 127.0.0.1:4000 (to access the UI from remote machines, change the bind address to 0.0.0.0, either in the Glowroot UI under Configuration > Web or directly in the admin.json file, and then restart JVM to take effect)

[trask]

Hi @FJGL, I'd like to understand why you are getting those WARN messages. I updated the WARN messages to capture more detail. Can you try out the latest snapshot release and then post the glowroot log file? Thanks. glowroot-agent-0.10.4-SNAPSHOT-dist.zip

[FJGL]

Here you are.

glowroot.2018-03-01.log

[trask]

Thanks, can you also post the JVM args from Glowroot > JVM > Environment?

[FJGL]

That's exactly what I was checking.
The problem is caused by the -Xhealthcenter that was in the boot arguments of websphere.
Thanks a lot.

Regards

[trask]

Thanks, I was able to reproduce using -Xhealthcenter and I made some improvements in the latest release glowroot-0.10.4-dist.zip to better log this condition, and to better support other agents (if you list glowroot agent first).