ConsulProcess#stop() is not shutting down ConsulLogHandler's logExecutor thread

[routp]

After calling ConsulProcess#stop() the consul process is stopped but the ConsulLogHandler's executor is still running. This holds the JVM from terminate as it is a non-daemon. There is a close() method in ConsulLogHandler but not getting invoked from ConsulProcess#stop(). Any way to call ConsulLogHandler#close explicitly to shutdown logExecutor?

These threads are getting added for each start of ConsulProcess.

2018-12-04 10:41:00
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):

"Attach Listener" #16 daemon prio=9 os_prio=0 tid=0x00007f88ec001000 nid=0x2f84 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"DestroyJavaVM" #15 prio=5 os_prio=0 tid=0x00007f892c00f000 nid=0x2ee1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"embedded-consul" #14 prio=5 os_prio=0 tid=0x00007f892cc92000 nid=0x2f36 waiting on condition [0x00007f88f41a9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076f6b3970> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- None

"embedded-consul" #12 prio=5 os_prio=0 tid=0x00007f892c9af000 nid=0x2f1c waiting on condition [0x00007f88f42c5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006ca16dff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- None

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f892c53f000 nid=0x2efb runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f892c539800 nid=0x2efa waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f892c537800 nid=0x2ef9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f892c536000 nid=0x2ef8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"Monitor Ctrl-Break" #5 daemon prio=5 os_prio=0 tid=0x00007f892c53a800 nid=0x2ef7 runnable [0x00007f8913e4d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x00000006ca246940> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x00000006ca246940> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)

Locked ownable synchronizers:
- None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f892c184000 nid=0x2ef6 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f892c150800 nid=0x2eef in Object.wait() [0x00007f89185d2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006ca247390> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000006ca247390> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

Locked ownable synchronizers:
- None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f892c14e000 nid=0x2eee in Object.wait() [0x00007f89186d3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006ca2475c0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006ca2475c0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Locked ownable synchronizers:
- None

"VM Thread" os_prio=0 tid=0x00007f892c144800 nid=0x2eea runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f892c024800 nid=0x2ee3 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f892c026000 nid=0x2ee4 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f892c028000 nid=0x2ee6 runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f892c029800 nid=0x2ee8 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f892c544000 nid=0x2efc waiting on condition

JNI global references: 253

[pszymczyk]

@routp Thanks for your contribution! I've just create PR resolving this issue.

[pszymczyk]

Resolved in 2.1.4 http://central.maven.org/maven2/com/pszymczyk/consul/embedded-consul/2.1.4/