Reconnect failure to collect stats

[rahulaga]

On JBoss 4.2.3 GA
Connection string: service:jmx:rmi:///jndi/rmi://HOST:PORT/jmxrmi

I believe the reconnection does happen as I stop seeing log message saying "Scheduling reconnect..."
However the error messages in the log are about bean not found:

[2014-09-25 16:47:27] FastJMX plugin: Failed javax.management.InstanceNotFoundException: java.lang:type=MemoryPool,name=CMS Perm Gen is not registered.
[2014-09-25 16:47:27] FastJMX plugin: Failed javax.management.InstanceNotFoundException: java.lang:type=MemoryPool,name=Par Survivor Space is not registered.
[2014-09-25 16:47:27] FastJMX plugin: Failed javax.management.InstanceNotFoundException: java.lang:type=MemoryPool,name=Code Cache is not registered.

What I noticed was that after I connect to jmx via jconsole and just look at those beans the errors magically got fixed and I started seeing successful collection!

I will continue to monitor how things works after next re-deploy.

[rahulaga]

Ok, I can confirm the problem. The reconnect is successful however it does not find the beans until I manually connect with jconsole. As soon as I use jconsole and poke around it starts working! Very strange behavior.

[bvarner]

It's going to be a few days (or weeks) before I get a chance to take a look at this in detail.

If you want to go poking around -- I'd appreciate the help! :-)

It's almost like the createPermutations isn't happening on reconnects...

[rahulaga]

So I figured out one thing that fixes this problem. If I do not merge the Jboss mbeans then this problem goes away!

Comment out these:
JAVA_OPTS="$JAVA_OPTS -Djboss.platform.mbeanserver"
JAVA_OPTS="$JAVA_OPTS -Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl"

[bvarner]

That keeps you from seeing the jboss internal mbeans though, right?

That would kinda stink, since one of the main reasons I use collectd is to get the app server pool metrics.

[rahulaga]

Correct. I can only get to basic JVM stats without that. So I think bug is probably with JBoss and not the collector.

[bvarner]

Did you experience this issue with the GenericJMX plugin as well?

[bvarner]

Oh, I forgot to mention -- do you get any type of error when it cannot read the beans after the redeploy?

[bvarner]

I just pushed an update that includes a TTL parameter on the Connection block. I wonder if forcibly re-connecting would get things to work. :-)

[rahulaga]

Yes, it was same on GenericJMX. The error I used to get is in the first message on this thread.
I'll have to try things out sometime. I realized that connecting via jconsole always works so maybe it is a problem with how the collector connects.

[bvarner]

Any luck with the TTL forcing it to re-identify mbeans after a deploy?

The idea being that for some reason bean registrations aren't being sent or recognized by the delegate server during a redeploy.

If the TTL expires, it forces the connection to close, flushes the beans from the list to collect, then re-initiates a new connection & Mbean discovery -- hopefully discovering the beans that have been deployed.

[rahulaga]

For now I have the jboss mbeans disabled as I'm just collecting basic jvm stats. Will let you know when I try it.

[bvarner]

I've been working with EAP 6.1 & Wildfly 8 this week, and have identified an issue (separate from this one) which I'll be addressing shortly.

Basically, the JBoss remoting service does not implement async connection notifications...

[bvarner]

Closing this issue due to inactivity.

Setting a TTL on the connection will force re-discovery upon reconnect. In my testing with JBoss EAP 5 and EAP 6, all beans are collected properly after a redeploy once the TTL expiration is reached.