New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

-javaagent: CleanupThread broken when discovery is enabled - 1.2.2 #155

Closed
ssevertson opened this Issue Jul 1, 2014 · 4 comments

Comments

Projects
None yet
4 participants
@ssevertson

ssevertson commented Jul 1, 2014

If discovery is enabled (the default), and Jolokia is deployed as a Java Agent, the JVM never shuts down cleanly. This appears to be due to the CleanupThread implementation looking only considering Daemon threads, but the multicast discovery thread is not a daemon thread.

Partial stack trace from a machine after the app has exited cleanly (Tomcat 6, in this case):

"Thread-5" prio=10 tid=0x00007fef24001000 nid=0x1246 runnable [0x00007fef44cd2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x00000000f5a928d8> (a sun.nio.ch.Util$2)
        - locked <0x00000000f5a928e8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f5a92890> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:341)
        at java.lang.Thread.run(Thread.java:745)

"Jolokia Agent Cleanup Thread" daemon prio=10 tid=0x00007fef401bc000 nid=0x1244 in Object.wait() [0x00007fef44ed4000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f5a94d70> (a org.jolokia.discovery.MulticastSocketListenerThread)
        at java.lang.Thread.join(Thread.java:1281)
        - locked <0x00000000f5a94d70> (a org.jolokia.discovery.MulticastSocketListenerThread)
        at java.lang.Thread.join(Thread.java:1355)
        at org.jolokia.jvmagent.CleanupThread.joinThreads(CleanupThread.java:110)
        at org.jolokia.jvmagent.CleanupThread.run(CleanupThread.java:60)

"Thread-3" prio=10 tid=0x00007fef401b2000 nid=0x1242 runnable [0x00007fef450d6000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        - locked <0x00000000f5a94df0> (a java.net.PlainDatagramSocketImpl)
        at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
        - locked <0x00000000f5a94df0> (a java.net.PlainDatagramSocketImpl)
        at java.net.DatagramSocket.receive(DatagramSocket.java:786)
        - locked <0x00000000f5a94e20> (a java.net.DatagramPacket)
        - locked <0x00000000f5a94e48> (a java.net.MulticastSocket)
        at org.jolokia.discovery.MulticastSocketListenerThread.receiveMessage(MulticastSocketListenerThread.java:113)
        at org.jolokia.discovery.MulticastSocketListenerThread.run(MulticastSocketListenerThread.java:70)

In the meantime, a simple workaround is to disable discovery on the Java Agent: "-javaagent:/path/to/jolokia-agent.jar=discoveryEnabled=false". With this flag in place, the discovery thread never starts, the CleanupThread works as expected, and the JVM can cleanly shut down.

@fujohnwang

This comment has been minimized.

fujohnwang commented Jul 8, 2014

encountered same problem too, the solution works, thanks

@rhuss rhuss added this to the 1.3.0 milestone Jul 8, 2014

@rhuss

This comment has been minimized.

Owner

rhuss commented Jul 8, 2014

Thanks for reporting this bug, will fix it for 1.3.0

@gyc567

This comment has been minimized.

gyc567 commented Mar 21, 2016

and this bug has been fixed?

@gyc567

This comment has been minimized.

gyc567 commented Mar 21, 2016

why i use the 1.3.2 jolokia jvm agent,I found that the class: org.jolokia.discovery.MulticastSocketListenerThread will remain a lot of references ,
and the jvm gc never clean it up ,and then cause OUT OF MEMORY ERROR.
anyone can tell me how to fix it ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment