-javaagent: CleanupThread broken when discovery is enabled - 1.2.2

[ssevertson]

If discovery is enabled (the default), and Jolokia is deployed as a Java Agent, the JVM never shuts down cleanly. This appears to be due to the CleanupThread implementation looking only considering Daemon threads, but the multicast discovery thread is not a daemon thread.

Partial stack trace from a machine after the app has exited cleanly (Tomcat 6, in this case):

"Thread-5" prio=10 tid=0x00007fef24001000 nid=0x1246 runnable [0x00007fef44cd2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x00000000f5a928d8> (a sun.nio.ch.Util$2)
        - locked <0x00000000f5a928e8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f5a92890> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:341)
        at java.lang.Thread.run(Thread.java:745)

"Jolokia Agent Cleanup Thread" daemon prio=10 tid=0x00007fef401bc000 nid=0x1244 in Object.wait() [0x00007fef44ed4000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f5a94d70> (a org.jolokia.discovery.MulticastSocketListenerThread)
        at java.lang.Thread.join(Thread.java:1281)
        - locked <0x00000000f5a94d70> (a org.jolokia.discovery.MulticastSocketListenerThread)
        at java.lang.Thread.join(Thread.java:1355)
        at org.jolokia.jvmagent.CleanupThread.joinThreads(CleanupThread.java:110)
        at org.jolokia.jvmagent.CleanupThread.run(CleanupThread.java:60)

"Thread-3" prio=10 tid=0x00007fef401b2000 nid=0x1242 runnable [0x00007fef450d6000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        - locked <0x00000000f5a94df0> (a java.net.PlainDatagramSocketImpl)
        at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
        - locked <0x00000000f5a94df0> (a java.net.PlainDatagramSocketImpl)
        at java.net.DatagramSocket.receive(DatagramSocket.java:786)
        - locked <0x00000000f5a94e20> (a java.net.DatagramPacket)
        - locked <0x00000000f5a94e48> (a java.net.MulticastSocket)
        at org.jolokia.discovery.MulticastSocketListenerThread.receiveMessage(MulticastSocketListenerThread.java:113)
        at org.jolokia.discovery.MulticastSocketListenerThread.run(MulticastSocketListenerThread.java:70)

In the meantime, a simple workaround is to disable discovery on the Java Agent: "-javaagent:/path/to/jolokia-agent.jar=discoveryEnabled=false". With this flag in place, the discovery thread never starts, the CleanupThread works as expected, and the JVM can cleanly shut down.

[fujohnwang]

encountered same problem too, the solution works, thanks

[rhuss]

Thanks for reporting this bug, will fix it for 1.3.0

[gyc567]

and this bug has been fixed?

[gyc567]

why i use the 1.3.2 jolokia jvm agent,I found that the class: org.jolokia.discovery.MulticastSocketListenerThread will remain a lot of references ,
and the jvm gc never clean it up ,and then cause OUT OF MEMORY ERROR.
anyone can tell me how to fix it ?