KAFKA-13187: Replace EasyMock / PowerMock with Mockito in DistributedHerderTest

[yashmayya]

• https://issues.apache.org/jira/browse/KAFKA-13187
• Original / abandoned PR for this issue is - Replace EasyMock/PowerMock with Mockito in DistributedHerderTest #11792.
• Mocking with explicit constructor fails with varargs parameter mockito/mockito#2601 is resolved in the current version of Mockito that we're using (4.11.0) so the workaround hack is no longer needed. -> the issue still exists on newer JDKs, a different workaround hack is used (by modifying the testing-only constructor for DistributedHerder)
• The current / new patch takes a slightly different approach from the original one. Using a combination of Mockito's strict stubs functionality and verifyNoMoreInteractions, we get some nice properties -
  • Any unused stubbings will lead to test failures
  • Tests fail early when stubbed methods are invoked with different arguments
  • Stubbed invocations are automatically verified and don't need explicit verification.
• The above properties allow us to make the test code significantly leaner and cleaner (as can be seen from the + / - SLOC in this diff).
• One big caveat with this approach is that the number of invocations for stubbed methods isn't automatically verified for us since Mockito stubbings are applicable any number of times by default (unlike in EasyMock where the default is a single invocation being stubbed).
• Hence, the older EasyMock way of stacking up expectations and then verifying them automatically took care of verifying the number of invocations for stubbed methods (assuming anyTimes() wasn't used in the expect calls).
• While for a lot of cases this really isn't required (for instance, verifying the number of calls to WorkerGroupMember::wakeup in large tests that involve multiple calls to DistributedHerder::tick), there are some cases where it is crucial to the test's coverage and this patch attempts to retain that coverage wherever it seemed important by using the times() verification mode explicitly.
• Only some minor refactoring was required for the DistributedHerder class itself -
  • The visibility of the herderExecutor field was bumped up for this specific test -

    kafka/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/distributed/DistributedHerderTest.java

    Line 3906 in ff390ab

    public void testThreadNames() {

  • The constructor used for testing was modified to use an array instead of varargs for the uponShutdown parameter in order to work around Mocking with explicit constructor fails with varargs parameter mockito/mockito#2601 here -

    kafka/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java

    Line 277 in 938fee2

    AutoCloseable... uponShutdown) {

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[yashmayya]

@C0urante @mimaison would you be able to take a look at this one? This effort was originally started 1 ½ years ago here and has had multiple ownership changes. I started working on this quite a while back myself but had to park it midway numerous times because it was taking way too long but it's finally been completed (or well, at least the tests all consistently pass and I'm hoping we've not lost coverage) 🙂

[yashmayya]

Hm I just noticed that the DistributedHerderTest is failing on the CI run (only for JDK 11) with:

org.mockito.exceptions.base.MockitoException: Unable to create mock instance of type 'DistributedHerder'
	at app//org.apache.kafka.connect.runtime.distributed.DistributedHerderTest.setUp(DistributedHerderTest.java:300)
	at java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566)
	at app//org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at app//org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at app//org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at app//org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
	at app//org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at app//org.mockito.internal.runners.DefaultInternalRunner$1$1.evaluate(DefaultInternalRunner.java:55)
	at app//org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at app//org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at app//org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at app//org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at app//org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at app//org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at app//org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at app//org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at app//org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at app//org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at app//org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at app//org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at app//org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at app//org.mockito.internal.runners.DefaultInternalRunner$1.run(DefaultInternalRunner.java:100)
	at app//org.mockito.internal.runners.DefaultInternalRunner.run(DefaultInternalRunner.java:107)
	at app//org.mockito.internal.runners.StrictRunner.run(StrictRunner.java:41)
	at app//org.mockito.junit.MockitoJUnitRunner.run(MockitoJUnitRunner.java:163)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
	at jdk.internal.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
	at java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: org.mockito.creation.instance.InstantiationException: 
Unable to create instance of 'DistributedHerder'.
Please ensure the target class has a constructor that matches these argument types: [org.apache.kafka.connect.runtime.distributed.DistributedConfig, org.apache.kafka.connect.runtime.Worker, java.lang.String, java.lang.String, org.apache.kafka.connect.storage.StatusBackingStore$MockitoMock$p8qyFkRn, org.apache.kafka.connect.storage.ConfigBackingStore$MockitoMock$4oJiFNI3, org.apache.kafka.connect.runtime.distributed.WorkerGroupMember, java.lang.String, org.apache.kafka.connect.runtime.rest.RestClient, org.apache.kafka.connect.runtime.MockConnectMetrics, org.apache.kafka.common.utils.MockTime, org.apache.kafka.connect.runtime.distributed.SampleConnectorClientConfigOverridePolicy, java.util.Collections$EmptyList, null, [Ljava.lang.AutoCloseable;] and executes cleanly.
	... 50 more
Caused by: java.lang.reflect.InvocationTargetException
	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.newInstance(InstrumentationMemberAccessor.java:198)
	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.newInstance(InstrumentationMemberAccessor.java:161)
	at org.mockito.internal.util.reflection.ModuleMemberAccessor.newInstance(ModuleMemberAccessor.java:42)
	at org.mockito.internal.creation.instance.ConstructorInstantiator.invokeConstructor(ConstructorInstantiator.java:70)
	at org.mockito.internal.creation.instance.ConstructorInstantiator.withParams(ConstructorInstantiator.java:53)
	at org.mockito.internal.creation.instance.ConstructorInstantiator.newInstance(ConstructorInstantiator.java:39)
	at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.doCreateMock(InlineDelegateByteBuddyMockMaker.java:364)
	at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.createMock(InlineDelegateByteBuddyMockMaker.java:334)
	at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.createMock(InlineByteBuddyMockMaker.java:56)
	at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:99)
	at org.mockito.internal.MockitoCore.mock(MockitoCore.java:88)
	at org.mockito.Mockito.mock(Mockito.java:2037)
	... 50 more
Caused by: java.lang.ClassCastException: Cannot cast [Ljava.lang.AutoCloseable; to java.lang.AutoCloseable
	at java.base/java.lang.Class.cast(Class.java:3605)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
	at java.base/java.lang.invoke.MethodHandleImpl$AsVarargsCollector.invokeWithArguments(MethodHandleImpl.java:578)
	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor$Dispatcher$ByteBuddy$aMere0wp.invokeWithArguments(Unknown Source)
	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.lambda$newInstance$0(InstrumentationMemberAccessor.java:191)
	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.newInstance(InstrumentationMemberAccessor.java:188)
	... 61 more

This is the same issue (mockito/mockito#2601) that was discovered in #11792 but supposed to be fixed in a newer version of Mockito (which we're already using). I'll take a closer look at why this is re-surfacing only on the JDK 11 build (and not on the other JDK builds; these tests were consistently passing for me locally with JDK 8).

Edit: I was able to reproduce this locally. The tests pass on JDK 8 but fail with the above issue on JDK 11 and JDK 17 as well (the CI run was skipping this test on JDK 17 because this bit in the build config wasn't removed -

kafka/build.gradle

Lines 413 to 418 in 3ba718e

	// Exclude PowerMock tests when running with Java 16 or newer until a version of PowerMock that supports the relevant versions is released
	// The relevant issues are https://github.com/powermock/powermock/issues/1094 and https://github.com/powermock/powermock/issues/1099
	if (JavaVersion.current().isCompatibleWith(JavaVersion.VERSION_16)) {
	testsToExclude.addAll([
	// connect tests
	"**/DistributedHerderTest.*",

[C0urante]

Thanks Yash, this is a truly Herculean effort and I really appreciate the work it took. Everything looks great, I've left a few small comments but I don't expect this to take more than one or two rounds before we can merge.

[yashmayya]

Thanks for the equally Herculean review Chris, your attention to detail is second to none 😄

I believe I've addressed all the review comments.

[C0urante]

LGTM, thanks again Yash!

[C0urante]

I should note that because this test makes heavy use of Mockito's strict stubbing feature, I've opened #14186 to enable reporting of unused stubbings in our CI builds. I've verified locally that there are no unused stubbings in this PR (./gradlew :connect:runtime:test --tests DistributedHerderTest does the trick), and any local build that uses the test task instead of unitTest should also report unused stubbings, so I don't believe it's necessary to block on #14186 before merging this.

Test failures in CI also appear unrelated. Merging...