MockitoJUnitRunner causes NPE when using @Mock on MockedStatic fields

[Cybermite]

check that

• The mockito message in the stacktrace have useful information, but it didn't help
• The problematic code (if that's possible) is copied here;
  Note that some configuration are impossible to mock via Mockito
• Provide versions (mockito / jdk / os / any other relevant information)
• Provide a Short, Self Contained, Correct (Compilable), Example of the issue
  (same as any question on stackoverflow.com)
• Read the contributing guide

Versions:

• mockito: 3.4.4
• junit: 4.12
• jdk: 1.8
• os: windows 10

Example:

1. MockitoJUnitRunnerWithMockedStaticTest
   • Reproduces the problem documented in this issue.
2. MockitoOpenMocksMockedStaticTest
   • Same tests but manually opens and closes the mocks (doesn't use the runner). Shows the tests are ran as expected.

Problem:

The MockitoJUnitRunner is causing a NullPointerException when the test class contains a @Mock instance field with a type of MockedStatic. This exception only occurs for tests that are ran after a prior test fails. If all the tests pass, there are no issues.

Stack Trace (Test 1):

java.lang.AssertionError: intentional failure
	at org.junit.Assert.fail(Assert.java:88)
	at org.mockito.example.MockitoJUnitRunnerWithMockedStaticTest.testName1(MockitoJUnitRunnerWithMockedStaticTest.java:40)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.mockito.internal.runners.DefaultInternalRunner$1$1.evaluate(DefaultInternalRunner.java:54)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.mockito.internal.runners.DefaultInternalRunner$1.run(DefaultInternalRunner.java:99)
	at org.mockito.internal.runners.DefaultInternalRunner.run(DefaultInternalRunner.java:105)
	at org.mockito.internal.runners.StrictRunner.run(StrictRunner.java:40)
	at org.mockito.junit.MockitoJUnitRunner.run(MockitoJUnitRunner.java:163)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)

org.mockito.exceptions.misusing.NotAMockException: Argument passed to Mockito.mockingDetails() should be a mock, but is an instance of class java.lang.Class!
	at org.mockito.internal.runners.DefaultInternalRunner$1$2.testFinished(DefaultInternalRunner.java:81)
	at org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:56)
	at org.junit.runner.notification.RunNotifier$7.notifyListener(RunNotifier.java:190)
	at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
	at org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:187)
	at org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:331)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.mockito.internal.runners.DefaultInternalRunner$1.run(DefaultInternalRunner.java:99)
	at org.mockito.internal.runners.DefaultInternalRunner.run(DefaultInternalRunner.java:105)
	at org.mockito.internal.runners.StrictRunner.run(StrictRunner.java:40)
	at org.mockito.junit.MockitoJUnitRunner.run(MockitoJUnitRunner.java:163)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)

Stack Trace (Test 2):

java.lang.NullPointerException
	at org.mockito.example.MockitoJUnitRunnerWithMockedStaticTest.testName2(MockitoJUnitRunnerWithMockedStaticTest.java:54)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.mockito.internal.runners.DefaultInternalRunner$1$1.evaluate(DefaultInternalRunner.java:54)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.mockito.internal.runners.DefaultInternalRunner$1.run(DefaultInternalRunner.java:99)
	at org.mockito.internal.runners.DefaultInternalRunner.run(DefaultInternalRunner.java:105)
	at org.mockito.internal.runners.StrictRunner.run(StrictRunner.java:40)
	at org.mockito.junit.MockitoJUnitRunner.run(MockitoJUnitRunner.java:163)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)

Investigation:

After some investigation, it appears the first test is failing twice (once in the test and once in the testFinished listener). This code:

mockito/src/main/java/org/mockito/internal/runners/DefaultInternalRunner.java

Lines 75 to 96 in 573bf0d

	public void testFinished(Description description)
	throws Exception {
	try {
	if (mockitoTestListener != null) {
	Mockito.framework()
	.removeListener(mockitoTestListener);
	mockitoTestListener.testFinished(
	new DefaultTestFinishedEvent(
	target,
	description.getMethodName(),
	failure));
	mockitoTestListener = null;
	}
	Mockito.validateMockitoUsage();
	} catch (Throwable t) {
	// In order to produce clean exception to the user we
	// need to fire test failure with the right description
	// Otherwise JUnit framework will report failure with
	// some generic test name
	notifier.fireTestFailure(new Failure(description, t));
	}
	}

Line 81 is where it fails the second time. If we navigate down the call hierarchy, it appears it fails because the static mock has already been cleaned up so the framework doesn't think it's a mock and fails with a NotAMockException. Since it fails there, it doesn't set the listener to null which will in-turn result in the mocks not being initialized on the next test that's ran (hits line 51):

mockito/src/main/java/org/mockito/internal/runners/DefaultInternalRunner.java

Lines 42 to 59 in 573bf0d

	public void evaluate() throws Throwable {
	AutoCloseable closeable;
	if (mockitoTestListener == null) {
	// get new test listener and add it to the framework
	mockitoTestListener = listenerSupplier.get();
	Mockito.framework().addListener(mockitoTestListener);
	// init annotated mocks before tests
	closeable = MockitoAnnotations.openMocks(target);
	} else {
	closeable = null;
	}
	try {
	base.evaluate();
	} finally {
	if (closeable != null) {
	closeable.close();
	}
	}

Since the mocks are not initialized for the second test, it ultimately causes the NullPointerException.

It's also causing the remaining tests after the first test failure to bounce back and forth between NullPointerException and NotAMockException. The reason for this appears to be because this failure object never gets reset after it's processed so it keeps restarting the chain of throwing the two exceptions back and forth.

Also, I did include a sample test that manually opens and closes the mocks (doesn't use MockitoJUnitRunner) and everything is working as expected. Using the try-with-resource works fine as well.

I did investigate further and noticed something related to this might have been addressed in this PR. It only updated the findStubbings method to skip the static mocks, though. Was the AllInvocationsFinder.find(...) method intentional left out of that or just an oversight?

[Cybermite]

Just tried the tests with using MockitoRule and it's working as expected as well.

[Cybermite]

@raphw this issue still exists in 3.4.6. Was the PR supposed to fix all of this or only a part of it? The linked example above still causes NPE. This one: MockitoJUnitRunnerWithMockedStaticTest

[raphw]

I was under the impression that I fixed it but I will take another look if it is not. I refactored your test and added it to the built but maybe I broke it in the process.

[raphw]

I just ran the test with the latest Mocktio and it works as expected. Are you sure that you are running the right version?

[Cybermite]

Yep, just doubled checked. It's using 3.4.6. Are you running my test class (without modification) or your refactored version of it? If you modified it, can you send me modified one and I'll double check it.

[raphw]

Copy pasted it as it is i to the release/3.x branch.

[Cybermite]

Okay, sorry for the confusion. When I was investigating this issue on July 25, I must have installed into locally and it got shoved into my local .m2 repo as 3.4.6 without the fix. Then when you fixed it and I updated the pom version to point to 3.4.6 it didn't redownload it because it was already in my local repo.

I deleted it from my local .m2 repo and redownloaded it from maven-central. Everything is good now. Thanks!

[raphw]

Perfect, Gradle does not work with snapshot versions, happened to me a few times, too.