Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stopping ReservedThreadExecutor may hang #4575

Closed
sbordet opened this issue Feb 15, 2020 · 4 comments
Closed

Stopping ReservedThreadExecutor may hang #4575

sbordet opened this issue Feb 15, 2020 · 4 comments
Projects
Jetty 9.4.27

Comments

@sbordet
Copy link
Contributor

sbordet commented Feb 15, 2020

Jetty version
10.0.x

Java version
13

Description
Running the test suite, test org.eclipse.jetty.embedded.ExampleServerTest may result in theJVM hanging during stop:

"main" #1 prio=5 os_prio=0 cpu=1217,68ms elapsed=268,04s tid=0x00007f88d801c800 nid=0x1a9f waiting on condition  [0x00007f88dde9d000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
	- parking to wait for  <0x00000000fd03bac8> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@13.0.2/SynchronousQueue.java:460)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@13.0.2/SynchronousQueue.java:361)
	at java.util.concurrent.SynchronousQueue.put(java.base@13.0.2/SynchronousQueue.java:873)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.offer(ReservedThreadExecutor.java:278)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.stop(ReservedThreadExecutor.java:292)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor.doStop(ReservedThreadExecutor.java:188)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:135)
	- locked <0x00000000fd195e98> (a java.lang.Object)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:185)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:208)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.doStop(QueuedThreadPool.java:201)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:135)
	- locked <0x00000000fe5f5a58> (a java.lang.Object)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:185)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:208)
	at org.eclipse.jetty.io.ClientConnector.doStop(ClientConnector.java:191)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:135)
	- locked <0x00000000fe6d2df8> (a java.lang.Object)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:185)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:208)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:135)
	- locked <0x00000000fe128548> (a java.lang.Object)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:185)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:208)
	at org.eclipse.jetty.client.HttpClient.doStop(HttpClient.java:250)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:135)
	- locked <0x00000000fe158e80> (a java.lang.Object)
	at org.eclipse.jetty.embedded.AbstractEmbeddedTest.stopClient(AbstractEmbeddedTest.java:56)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@13.0.2/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@13.0.2/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@13.0.2/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@13.0.2/Method.java:567)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:111)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptAfterEachMethod(TimeoutExtension.java:95)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor$$Lambda$409/0x0000000800cc9440.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall$$Lambda$159/0x0000000800bb9c40.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$$Lambda$295/0x0000000800bf9c40.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:464)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$17(ClassBasedTestDescriptor.java:454)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor$$Lambda$267/0x0000000800bea040.invokeAfterEachMethod(Unknown Source)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$9(TestMethodTestDescriptor.java:228)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$407/0x0000000800cc9c40.invoke(Unknown Source)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$12(TestMethodTestDescriptor.java:256)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$408/0x0000000800cc9040.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:256)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$406/0x0000000800cc9840.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@13.0.2/ArrayList.java:1507)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAllAfterMethodsOrCallbacks(TestMethodTestDescriptor.java:255)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterEachMethods(TestMethodTestDescriptor.java:226)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$237/0x0000000800be1c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$236/0x0000000800be1840.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$235/0x0000000800be1440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$241/0x0000000800be2c40.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@13.0.2/ArrayList.java:1507)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$237/0x0000000800be1c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$236/0x0000000800be1840.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$235/0x0000000800be1440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$241/0x0000000800be2c40.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@13.0.2/ArrayList.java:1507)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$237/0x0000000800be1c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$236/0x0000000800be1840.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$235/0x0000000800be1440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:229)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:197)
	at org.junit.platform.launcher.core.DefaultLauncher$$Lambda$202/0x0000000800bc6840.accept(Unknown Source)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:211)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:191)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:127)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0,32ms elapsed=268,02s tid=0x00007f88d8436800 nid=0x1aaa waiting on condition  [0x00007f88bc3e3000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@13.0.2/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@13.0.2/Reference.java:241)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@13.0.2/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0,17ms elapsed=268,02s tid=0x00007f88d8438800 nid=0x1aab in Object.wait()  [0x00007f88bc2e2000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@13.0.2/Native Method)
	- waiting on <0x00000000fca00fb0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@13.0.2/ReferenceQueue.java:155)
	- locked <0x00000000fca00fb0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@13.0.2/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@13.0.2/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0,69ms elapsed=268,02s tid=0x00007f88d843e800 nid=0x1aac runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=2581,37ms elapsed=268,02s tid=0x00007f88d8440800 nid=0x1aad waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #8 daemon prio=9 os_prio=0 cpu=1128,98ms elapsed=268,02s tid=0x00007f88d8442800 nid=0x1aae waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #9 daemon prio=9 os_prio=0 cpu=40,74ms elapsed=268,02s tid=0x00007f88d8444800 nid=0x1aaf runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #10 daemon prio=8 os_prio=0 cpu=0,83ms elapsed=268,01s tid=0x00007f88d8453000 nid=0x1ab0 in Object.wait()  [0x00007f88a6f8a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@13.0.2/Native Method)
	- waiting on <0x00000000fca05280> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@13.0.2/ReferenceQueue.java:155)
	- locked <0x00000000fca05280> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@13.0.2/CleanerImpl.java:148)
	at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)
	at jdk.internal.misc.InnocuousThread.run(java.base@13.0.2/InnocuousThread.java:134)

"Service Thread" #12 daemon prio=9 os_prio=0 cpu=1,45ms elapsed=267,89s tid=0x00007f88d87c1000 nid=0x1ab2 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"surefire-forkedjvm-command-thread" #13 daemon prio=5 os_prio=0 cpu=11,65ms elapsed=267,84s tid=0x00007f88d87fb800 nid=0x1ab4 runnable  [0x00007f88a6b85000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes(java.base@13.0.2/Native Method)
	at java.io.FileInputStream.read(java.base@13.0.2/FileInputStream.java:273)
	at java.io.BufferedInputStream.fill(java.base@13.0.2/BufferedInputStream.java:245)
	at java.io.BufferedInputStream.read(java.base@13.0.2/BufferedInputStream.java:264)
	- locked <0x00000000fca02e38> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.readInt(java.base@13.0.2/DataInputStream.java:392)
	at org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:113)
	at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:383)
	at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

"client-15" #15 prio=5 os_prio=0 cpu=6,39ms elapsed=267,04s tid=0x00007f88d8dfa000 nid=0x1ac8 waiting on condition  [0x00007f88a5c71000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
	- parking to wait for  <0x00000000fe337bb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@13.0.2/LockSupport.java:235)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@13.0.2/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:224)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:932)
	at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

"client-16" #16 prio=5 os_prio=0 cpu=6,48ms elapsed=267,04s tid=0x00007f88d8dfb000 nid=0x1ac9 waiting on condition  [0x00007f88a5b70000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
	- parking to wait for  <0x00000000fe337bb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@13.0.2/LockSupport.java:235)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@13.0.2/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:224)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:932)
	at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

"client-17" #17 prio=5 os_prio=0 cpu=50,99ms elapsed=267,04s tid=0x00007f88d8dfd000 nid=0x1aca waiting on condition  [0x00007f88a5a6f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
	- parking to wait for  <0x00000000fe337bb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@13.0.2/LockSupport.java:235)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@13.0.2/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:224)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:932)
	at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

"client-18" #18 prio=5 os_prio=0 cpu=1,23ms elapsed=267,04s tid=0x00007f88d8dfe800 nid=0x1acb waiting on condition  [0x00007f88a596e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
	- parking to wait for  <0x00000000fe337bb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@13.0.2/LockSupport.java:235)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@13.0.2/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:224)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:932)
	at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

"client-19" #19 prio=5 os_prio=0 cpu=9,18ms elapsed=267,04s tid=0x00007f88d8e00800 nid=0x1acc waiting on condition  [0x00007f88a586d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
	- parking to wait for  <0x00000000fe337bb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@13.0.2/LockSupport.java:235)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@13.0.2/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:224)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:932)
	at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

"client-20" #20 prio=5 os_prio=0 cpu=0,86ms elapsed=267,04s tid=0x00007f88d8e02000 nid=0x1acd waiting on condition  [0x00007f88a576c000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
	- parking to wait for  <0x00000000fd03bf18> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@13.0.2/SynchronousQueue.java:460)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@13.0.2/SynchronousQueue.java:361)
	at java.util.concurrent.SynchronousQueue.put(java.base@13.0.2/SynchronousQueue.java:873)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.offer(ReservedThreadExecutor.java:278)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor.tryExecute(ReservedThreadExecutor.java:220)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:320)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:823)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:945)
	at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

"client-21" #21 prio=5 os_prio=0 cpu=0,80ms elapsed=267,04s tid=0x00007f88d8e04000 nid=0x1ace waiting on condition  [0x00007f88a566b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
	- parking to wait for  <0x00000000fe337bb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@13.0.2/LockSupport.java:235)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@13.0.2/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:224)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:932)
	at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

"client-22" #22 prio=5 os_prio=0 cpu=0,58ms elapsed=267,04s tid=0x00007f88d8e06000 nid=0x1acf waiting on condition  [0x00007f88a556a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
	- parking to wait for  <0x00000000fe337bb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@13.0.2/LockSupport.java:235)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@13.0.2/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:224)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:932)
	at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

What above is trying to stop HttpClient, and its QueuedThreadPool, but the ReservedThreadExecutor seems to hang waiting for a thread, although there seems to be threads available in the pool (those client-* threads).
@sbordet
Copy link
Contributor Author

sbordet commented Feb 15, 2020

@gregw ideas?

@gregw
Copy link
Contributor

gregw commented Feb 15, 2020

@sbordet I think the problem is that a reserved thread can be stopped either by: it sees that isRunning is false; or it is passed a poison pill.
This stack looks like we are trying to give the thread the poison pill, but it has already seen that isRunning is false and doesn't poll the synchronous queue.

The solution is that we should only ever stop a reserved thread with a poison pill, so that way it will always be polling the synchronous queue and we will never block forever handing over a task.

@gregw
Copy link
Contributor

gregw commented Feb 15, 2020

This is likely to be a problem in jetty-9 also!

@gregw gregw added this to To do in Jetty 9.4.27 via automation Feb 15, 2020
@gregw gregw mentioned this issue Feb 16, 2020
Merged
@gregw
Copy link
Contributor

gregw commented Feb 16, 2020

See #4576

sbordet added a commit that referenced this issue Feb 16, 2020
@sbordet
Merge pull request #4576 from eclipse/jetty-9.4.x-4575-stopping-reser…
7320b1d 
…ved-threads

Fixes #4575 Stopping Reserved Thread
Jetty 9.4.27 automation moved this from To do to Done Feb 16, 2020
@sbordet sbordet mentioned this issue Feb 27, 2020
Closed
@joakime joakime mentioned this issue Mar 4, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
No open projects
Jetty 9.4.27
  
Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gregw @sbordet