Why do Mock Testing Frameworks and RV-Monitor interact negatively?

[owolabileg]

Solving this problem will likely help us run our tools on more non-trivial open-source projects.

In a good number of open-source applications that I have tried, tests pass 100% without monitoring but, when monitoring is enabled, I see test failures. I inspected some of them and realize that it is due to rv-monitor + mocking interactions. I have noticed the same thing for at least 5 unrelated mocking frameworks and I feel that is unlikely to be a random bug. I have also seen this in a good number of projects.

Here is a sample stack trace for test failures in a project (caelum.vraptor) that happens only after I start monitoring:

containsMessageShouldReturnFalseWhenExpectedMessageDontExists(br.com.caelum.vraptor.util.test.MockValidatorTest)  Time elapsed: 0.157 sec  <<< ERROR!
java.lang.VerifyError: (class: br/com/caelum/vraptor/util/test/MockValidator$$FastClassByMockitoWithCGLIB$$a16639d0, method: newInstance signature: (I[Ljava/lang/Object;)Ljava/lang/Object;) Accessing value\
 from uninitialized register 3
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532)
        at java.lang.Class.getConstructor0(Class.java:2842)
        at java.lang.Class.getDeclaredConstructor(Class.java:2053)
        at br.com.caelum.vraptor.util.test.MockValidatorTest.containsMessageShouldReturnFalseWhenExpectedMessageDontExists(MockValidatorTest.java:30)

containsMessageShouldReturnTrueWhenExpectedMessageExists(br.com.caelum.vraptor.util.test.MockValidatorTest)  Time elapsed: 0.025 sec  <<< ERROR!
java.lang.ExceptionInInitializerError: null
        at com.runtimeverification.rvmonitor.java.rt.ViolationRecorder.makeRelevantList(ViolationRecorder.java:92)
        at com.runtimeverification.rvmonitor.java.rt.ViolationRecorder.getRelevantStack(ViolationRecorder.java:46)
        at com.runtimeverification.rvmonitor.java.rt.ViolationRecorder.getLineOfCode(ViolationRecorder.java:54)
        at mop.Serializable_UIDRawMonitor.event_staticinit(JavaMOPAgentRuntimeMonitor.java:24075)
        at mop.JavaMOPAgentRuntimeMonitor.Serializable_UID_staticinitEvent(JavaMOPAgentRuntimeMonitor.java:49334)
        at mop.JavaMOPAgentMonitorAspect.ajc$after$mop_JavaMOPAgentMonitorAspect$305$cea3d0ce(JavaMOPAgentMonitorAspect.aj:1777)
        at br.com.caelum.vraptor.validator.Message.<clinit>(Message.java:1)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:190)
        at br.com.caelum.vraptor.validator.Message$$EnhancerByMockitoWithCGLIB$$7ff0bdae.CGLIB$STATICHOOK58(<generated>:1)
        at br.com.caelum.vraptor.validator.Message$$EnhancerByMockitoWithCGLIB$$7ff0bdae.<clinit>(<generated>:1)
        at sun.reflect.GeneratedSerializationConstructorAccessor103.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:45)
        at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:73)
        at org.mockito.internal.creation.jmock.ClassImposterizer.createProxy(ClassImposterizer.java:111)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:51)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:54)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:45)
        at org.mockito.Mockito.mock(Mockito.java:921)
    at org.mockito.Mockito.mock(Mockito.java:816)
        at br.com.caelum.vraptor.util.test.MockValidatorTest.containsMessageShouldReturnTrueWhenExpectedMessageExists(MockValidatorTest.java:35)

containsMessageShouldReturnTrueWhenExpectedMessageWithParamsExists(br.com.caelum.vraptor.util.test.MockValidatorTest)  Time elapsed: 0 sec  <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class br.com.caelum.vraptor.validator.Message$$EnhancerByMockitoWithCGLIB$$7ff0bdae
        at sun.reflect.GeneratedSerializationConstructorAccessor103.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:45)
        at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:73)
        at org.mockito.internal.creation.jmock.ClassImposterizer.createProxy(ClassImposterizer.java:111)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:51)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:54)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:45)
        at org.mockito.Mockito.mock(Mockito.java:921)
        at org.mockito.Mockito.mock(Mockito.java:816)
        at br.com.caelum.vraptor.util.test.MockValidatorTest.containsMessageShouldReturnTrueWhenExpectedMessageWithParamsExists(MockValidatorTest.java:45)

Here is the content of the BaseAspect.aj file that use in building the agent:

package mop;
public aspect BaseAspect {
  pointcut notwithin() :
  !within(sun..*) &&
  !within(java..*) &&
  !within(javax..*) &&
  !within(com.sun..*) &&
  !within(org.dacapo.harness..*) &&
  !within(org.apache.commons..*) &&
  !within(org.apache.geronimo..*) &&
  !within(net.sf.cglib..*) &&
  !within(mop..*) &&
  !within(javamoprt..*) &&
  !within(rvmonitorrt..*) &&
  !within(org.junit..*) &&
  !within(junit..*) &&
  !within(java.lang.Object) &&
  !within(com.runtimeverification..*) &&
  !within(org.apache.maven.surefire..*) &&
  !within(org.mockito..*) &&
  !within(org.powermock..*) &&
  !within(org.easymock..*) &&
  !within(com.mockrunner..*) &&
  !within(org.jmock..*);
}

The subject program is: https://github.com/caelum/vraptor.git 6a4f5e9df9a5d481cc4e44b820e19ad4617bc227

and here is the relevant relevant part of the vraptor's pom.xml after adding javamop agent to it:

<build>
<plugins>
  ....
  <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
    <version>2.16</version>
        <configuration>
          <argLine>-javaagent:javamop-agent.jar</argLine>
        </configuration>
      </plugin>
    </plugins>
</build>

where javamop-agent.jar is the agent built using the following command:

javamop -agent -n javamop-agent.jar -usedb -baseaspect BaseAspect.aj

(please ensure that the BaseAspect.aj has identical contents as the one I posted above)

[pdaian]

@owolabileg thanks for reporting! I agree we need to work with mock frameworks. I will look into it and keep you updated. This seems to me like an AspectJ-related issue so perhaps I will also need to contact AspectJ developers (eg - does not seem like a very trivial bug, it looks like the classloader is failing for some reason, either due to bad bytecode changes or something wrong in the classloader hooks).