Mockito 2.6.4 hangs on JDK 1.8.0_31 #892

Stephan202 · 2017-01-20T11:32:20Z

We just upgraded to Mockito 2.6.4. When we run our tests on Travis CI without the oracle-java8-installer, JDK 1.8.0_31 is used. We then have two threads hanging in the following state:

"pool-1-thread-2" #24 prio=5 os_prio=0 tid=0x00007fcd208ca800 nid=0x27b3 in Object.wait() [0x00007fcc8c1a1000]
   java.lang.Thread.State: RUNNABLE
        at net.bytebuddy.description.type.TypeDescription$Generic.<clinit>(TypeDescription.java:319)
        at net.bytebuddy.ByteBuddy.subclass(ByteBuddy.java:363)
        at net.bytebuddy.ByteBuddy.subclass(ByteBuddy.java:334)
        at net.bytebuddy.ByteBuddy.subclass(ByteBuddy.java:231)
        at org.mockito.internal.creation.bytebuddy.SubclassBytecodeGenerator.mockClass(SubclassBytecodeGenerator.java:66)
        at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator$1.call(TypeCachingBytecodeGenerator.java:35)
        at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator$1.call(TypeCachingBytecodeGenerator.java:32)
        at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:138)
        at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:346)
        at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:161)
        - locked <0x0000000689673668> (a java.lang.Class for com.microsoft.windowsazure.services.servicebus.ServiceBusContract)
        at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:355)
        at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator.mockClass(TypeCachingBytecodeGenerator.java:30)
        at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMockType(SubclassByteBuddyMockMaker.java:71)
        at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:42)
        at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:26)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:65)
        at org.mockito.Mockito.mock(Mockito.java:1681)
        at org.mockito.Mockito.mock(Mockito.java:1594)

The issue reported here does not reproduce with Mockito 2.6.3. It also does not reproduce with JDK 1.8.0_121. I assume it's caused by #891.

(I know upgrading our JDK would be "better", but if at all possible I'd like to see this fixed anyway, because using the oracle-java8-installer introduces a dependency on the Oracle website, and we've seen it being down from time-to-time, thus affecting our build stability.)

CC @raphw. (If you want me to file such tickets directly against Byte Buddy, let me know.)

The text was updated successfully, but these errors were encountered:

bric3 · 2017-01-20T12:18:00Z

@Stephan202 Thanks for filling the issue. There's plenty of bugs with JDK8 prior to u45 that affected Mockito. The first working version of the JDK is u45 it we don't support earlier buggy version of the JDK. This issue is different though, I don't know if there's enough life throughput for this bug, especially given the issue is resolved with later version of the JDK.

raphw · 2017-01-20T12:50:27Z

Do you have some special setup in your class loaders? I wonder how this could even relate to the update, there should not be a race to loading these classes. Also, nowhere in Byte Buddy, Object::wait is called.

raphw · 2017-01-20T13:12:28Z

Travis runs on a newer version for the non-dockerized build: travis-ci/travis-ci#3259 (comment) - It is really a pitty how far behind Travis is in this regard, but I am afraid there is little we can do here.

bric3 · 2017-01-20T13:13:12Z

@Stephan202 Could it be related to Azure libraries ? I saw a com.microsoft.windowsazure.services.servicebus.ServiceBusContract ?

bric3 · 2017-01-20T13:15:01Z

@raphw Yes @travis-ci is disappointing in this regard...

Stephan202 · 2017-01-21T10:04:43Z

@bric3, you were onto something. When I disable the two tests that mock ServiceBusContract, there rest of the build (821 mock statements across 167 test classes) passes fine. Looking at ServiceBusContract (we're using com.microsoft.azure:azure-servicebus:0.9.7), the only "interesting" thing about it is that it is a recursive type:

public interface ServiceBusContract extends
        JerseyFilterableService<ServiceBusContract> {

    ...

}

@raphw, might that be related?

raphw · 2017-01-21T10:44:53Z

Hey Stephan, you might be on to something, even though Byte Buddy resolves generic types lazily to support generic types. I will have a look and see if I can reproduce something.

raphw · 2017-01-21T11:22:27Z

@Stephan202 I tried to reproduce your bug by running a u31 version, mocking the ServiceBusContract from multiple threads. I do not experience a lock.

Could you:

Check what monitor is hold in your setup from the threads in question?
Attempt a minimal reproduction of your bug?

I wonder if we should not lock the class for the cache but be a bit more granular by locking the class' loader instead. We might be competing with other threads where a custom class loader locks the class as a class loader lock. In the end, we do not really know how they are implemented. For investigating this, knowing at least (1) would be very helpful.

Stephan202 · 2017-01-21T16:44:54Z

@raphw, I created a Gist with a reproduction case: https://gist.github.com/Stephan202/2a1650945c2c633217d4f30f89e4412e (the .java files should be in src/test/java/com/example/, but Github Gist doesn't allow me to make that explicit, unfortunately.)

For this reproduction case, the full thread details for the relevant two threads are:

"pool-1-thread-1" #23 prio=5 os_prio=0 tid=0x00007f20cc735800 nid=0x90f in Object.wait() [0x00007f2066a20000]
   java.lang.Thread.State: RUNNABLE
        at net.bytebuddy.description.type.TypeDescription$Generic.<clinit>(TypeDescription.java:319)
        at net.bytebuddy.ByteBuddy.subclass(ByteBuddy.java:363)
        at net.bytebuddy.ByteBuddy.subclass(ByteBuddy.java:334)
        at net.bytebuddy.ByteBuddy.subclass(ByteBuddy.java:231)
        at org.mockito.internal.creation.bytebuddy.SubclassBytecodeGenerator.mockClass(SubclassBytecodeGenerator.java:66)
        at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator$1.call(TypeCachingBytecodeGenerator.java:35)
        at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator$1.call(TypeCachingBytecodeGenerator.java:32)
        at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:138)
        at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:346)
        at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:161)
        - locked <0x0000000688cc4208> (a java.lang.Class for com.microsoft.windowsazure.services.servicebus.ServiceBusContract)
        at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:355)
        at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator.mockClass(TypeCachingBytecodeGenerator.java:30)
        at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMockType(SubclassByteBuddyMockMaker.java:71)
        at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:42)
        at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:26)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:65)
        at org.mockito.Mockito.mock(Mockito.java:1681)
        at org.mockito.Mockito.mock(Mockito.java:1594)
        at com.example.FirstTest.test(FirstTest.java:13)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
        at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
        at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
        at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
        at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-2" #24 prio=5 os_prio=0 tid=0x00007f20cc737000 nid=0x910 in Object.wait() [0x00007f2066920000]
   java.lang.Thread.State: RUNNABLE
        at net.bytebuddy.description.type.TypeDescription$AbstractBase.asGenericType(TypeDescription.java:6577)
        at net.bytebuddy.ByteBuddy.subclass(ByteBuddy.java:366)
        at net.bytebuddy.ByteBuddy.subclass(ByteBuddy.java:334)
        at net.bytebuddy.ByteBuddy.subclass(ByteBuddy.java:231)
        at org.mockito.internal.creation.bytebuddy.SubclassBytecodeGenerator.mockClass(SubclassBytecodeGenerator.java:66)
        at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator$1.call(TypeCachingBytecodeGenerator.java:35)
        at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator$1.call(TypeCachingBytecodeGenerator.java:32)
        at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:138)
        at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:346)
        at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:161)
        - locked <0x00000006892314a0> (a java.lang.Class for com.microsoft.windowsazure.services.servicebus.models.BrokeredMessage)
        at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:355)
        at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator.mockClass(TypeCachingBytecodeGenerator.java:30)
        at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMockType(SubclassByteBuddyMockMaker.java:71)
        at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:42)
        at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:26)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:65)
        at org.mockito.Mockito.mock(Mockito.java:1681)
        at org.mockito.Mockito.mock(Mockito.java:1594)
        at com.example.SecondTest.test(SecondTest.java:12)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
        at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
        at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
        at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
        at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Fix TypeCache dead lock Should fix #892

Stephan202 · 2017-01-23T20:07:42Z

@raphw, thanks! I've tested with version 2.6.8 and can confirm the fix works.

mockitoguy · 2017-01-24T15:15:20Z

This bug was really tricky. @raphw and @bric3 THANK you for being super fast in reacting to issues reported by our users. It's a pleasure working on open source together 🥇

@Stephan202, thank you for confirming the bug is fixed and reporting this issue diligently. It is very helpful to us when we get comprehensive bug reports such as this one.

raphw mentioned this issue Jan 23, 2017

Fix TypeCache dead lock #902

Merged

raphw self-assigned this Jan 23, 2017

raphw added the in progress label Jan 23, 2017

bric3 closed this as completed in #902 Jan 23, 2017

bric3 added a commit that referenced this issue Jan 23, 2017

Merge pull request #902 from mockito/dead-lock

ec71d3b

Fix TypeCache dead lock Should fix #892

bric3 removed the in progress label Jan 23, 2017

mockitoguy mentioned this issue Feb 12, 2017

Separate concise release notes file mockito/shipkit#9

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mockito 2.6.4 hangs on JDK 1.8.0_31 #892

Mockito 2.6.4 hangs on JDK 1.8.0_31 #892

Stephan202 commented Jan 20, 2017

bric3 commented Jan 20, 2017

raphw commented Jan 20, 2017 •

edited

raphw commented Jan 20, 2017

bric3 commented Jan 20, 2017

bric3 commented Jan 20, 2017

Stephan202 commented Jan 21, 2017

raphw commented Jan 21, 2017

raphw commented Jan 21, 2017

Stephan202 commented Jan 21, 2017

Stephan202 commented Jan 23, 2017

mockitoguy commented Jan 24, 2017

Mockito 2.6.4 hangs on JDK 1.8.0_31 #892

Mockito 2.6.4 hangs on JDK 1.8.0_31 #892

Comments

Stephan202 commented Jan 20, 2017

bric3 commented Jan 20, 2017

raphw commented Jan 20, 2017 • edited

raphw commented Jan 20, 2017

bric3 commented Jan 20, 2017

bric3 commented Jan 20, 2017

Stephan202 commented Jan 21, 2017

raphw commented Jan 21, 2017

raphw commented Jan 21, 2017

Stephan202 commented Jan 21, 2017

Stephan202 commented Jan 23, 2017

mockitoguy commented Jan 24, 2017

raphw commented Jan 20, 2017 •

edited