Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mutation analysis does not work for Mockito #308

Open
LuisaLiu opened this issue Mar 11, 2020 · 15 comments
Open

Mutation analysis does not work for Mockito #308

LuisaLiu opened this issue Mar 11, 2020 · 15 comments

Comments

@LuisaLiu
Copy link

@LuisaLiu LuisaLiu commented Mar 11, 2020

@rjust,hi~
First I used the Randoop tool to generate the test cases. Then I executed run_mutation.pl on 12f、14f of Mockito, the failed information like this:

update.all.tests:

mutation.test:
[echo] Running mutation analysis ...
[junit] MAJOR: Mutation analysis enabled

BUILD FAILED
/home/liu/defects4j/framework/projects/defects4j.build.xml:223: java.lang.RuntimeException: java.lang.ClassNotFoundException: RegressionTest0
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask$MajorResultFormatterTestSuite.addError(JUnitTask.java:3449)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:512)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.executeInVM(JUnitTask.java:1478)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute(JUnitTask.java:872)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.executeOrQueue(JUnitTask.java:1972)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute1(JUnitTask.java:824)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute(JUnitTask.java:2370)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:392)
at org.apache.tools.ant.Target.performTasks(Target.java:413)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
at org.apache.tools.ant.Main.runBuild(Main.java:811)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
Caused by: java.lang.ClassNotFoundException: RegressionTest0
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.tools.ant.AntClassLoader.findBaseClass(AntClassLoader.java:1387)
at org.apache.tools.ant.AntClassLoader.loadClass(AntClassLoader.java:1080)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:376)
... 21 more

Total time: 17 seconds

And then I also executed command on other version of Mockito, their failed information was the same as described above .

@rjust

This comment has been minimized.

Copy link
Owner

@rjust rjust commented Mar 11, 2020

Hi @LuisaLiu,

Can you please provide:

  1. The test suite archive with the generated Randoop tests
  2. The command that you executed (invocation of the run_mutation.pl script)

Thanks,
René

@LuisaLiu

This comment has been minimized.

Copy link
Author

@LuisaLiu LuisaLiu commented Mar 11, 2020

Hi, @rjust

1.This is the test suite that Randoop generated for Mockito(12f、14f) :
randoop_gen_tests.zip

2.This is the command I used:
perl run_mutation.pl -p Mockito -d /home/liu/bishe_test/Randoop-Test/Mockito-test-fixed/Mockito/randoop/1 -o /home/liu/bishe_test/Randoop-Test/Mockito-test-fixed/Mockito/randoop/mutation

Before I did the mutation analysis, I also executed the fix_test_suite.pl script to fixed the test suite.

Thanks.

@rjust

This comment has been minimized.

Copy link
Owner

@rjust rjust commented Mar 13, 2020

Hi @LuisaLiu,

I could reproduce this issue. I'll take a closer look and let you know.

Best,
René

@LuisaLiu

This comment has been minimized.

Copy link
Author

@LuisaLiu LuisaLiu commented Mar 13, 2020

Hi, @rjust

Ok. Thank you very much for your help!

@LuisaLiu LuisaLiu closed this Mar 13, 2020
@LuisaLiu LuisaLiu reopened this Mar 13, 2020
@rjust

This comment has been minimized.

Copy link
Owner

@rjust rjust commented Mar 16, 2020

A quick update on this: the mutation branch should fix this issue. Pending further testing (CI and locally), we should be able to merge this change into master soon.

Also, I noted that two of the generated tests (that you shared) fail under mutation but not under test. This can happen because of classloader differences. You might need to exclude such tests to have stable results.

@LuisaLiu

This comment has been minimized.

Copy link
Author

@LuisaLiu LuisaLiu commented Mar 16, 2020

Hi, @rjust

I will try to do mutation analysis on this branch later, and thank you very much for your reminding!

@rjust

This comment has been minimized.

Copy link
Owner

@rjust rjust commented Mar 22, 2020

Hi @LuisaLiu, does the mutation branch work for you? All tests have passed and we are ready to merge the changes, but I wanted to check whether they address all of your issues.

Thanks,
René

@LuisaLiu

This comment has been minimized.

Copy link
Author

@LuisaLiu LuisaLiu commented Mar 23, 2020

Hi @rjust

Thank you for your attention to my problem. I tried to do mutation analysis on mutation branch,but there was an error in the first step.
I followed the tutorial to run ./init.sh first, and then set the PATH.

The error report is as follows:

Setting up utility programs ...
Downloading https://github.com/jose/build-analyzer/releases/download/v0.0.1/build-analyzer-0.0.1.jar
Downloaded https://github.com/jose/build-analyzer/releases/download/v0.0.1/build-analyzer-0.0.1.jar

Defects4J successfully initialized.
liu@ubuntu:/defects4j-mutation$ export PATH=$PATH:/home/liu/defects4j-mutation/framework/bin
liu@ubuntu:
/defects4j-mutation$ defects4j info -p Lang
Couldn't find project repositories! Did you (re)run 'defects4j/init.sh'?

Compilation failed in require at /home/liu/defects4j-mutation/framework/bin/d4j/d4j-info line 57.
BEGIN failed--compilation aborted at /home/liu/defects4j-mutation/framework/bin/d4j/d4j-info line 57.
Compilation failed in require at /home/liu/defects4j-mutation/framework/bin/defects4j line 155.

I don't know if there is a problem with my experimental environment and it hasn't been solved yet. Do you have any comments?

Thanks!

@rjust

This comment has been minimized.

Copy link
Owner

@rjust rjust commented Mar 23, 2020

Hi @LuisaLiu,

I just executed the following commands and everything seems fine on my end:

git clone https://github.com/rjust/defects4j
cd defects4j
git checkout mutation
./init.sh
./framework/bin/defects4j info -pLang

Can you please run the above commands and, if the problem persists, provide the full output of the init.sh script?

Thanks,
René

@LuisaLiu

This comment has been minimized.

Copy link
Author

@LuisaLiu LuisaLiu commented Mar 24, 2020

Hi, @rjust

This happened a few days ago when I used the mutation branch: the initialization was very slow, and it stopped at the “Setting up project repositories”, so I replaced the file with the old "defects4j-repos.zip". And then the initialization works, but there's the problem I talked about last time.

Today, I want to do it again according to your steps, but still stuck in the initialization step. The initialization situation of today's day is like this:

liu@ubuntu:~/defects4j$ ./init.sh
Setting up project repositories ...
retrying curl https://defects4j.org/downloads/defects4j-repos.zip
Warning: Illegal date format for -z, --timecond (and not a file name).
Warning: Disabling time condition. See curl_getdate(3) for valid date syntax.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
33 418M 33 141M 0 0 17114 0 7:07:50 2:24:14 4:43:36

Maybe it is because I download the resources of foreign websites in China, the download speed will be limited. Now I only to wait for this step to succeed.

rjust added a commit that referenced this issue Mar 24, 2020
@rjust

This comment has been minimized.

Copy link
Owner

@rjust rjust commented Mar 24, 2020

Hi @LuisaLiu,

Hopefully, the initialization completes soon for you. For future testing/debugging, you should be able to rerun the init script after switching branches or pulling in new changes (without cloning from scratch). The init script will only download files if necessary.

Please let me know whether the mutation analysis works for you, after successfully initializing your local clone.

Thanks,
René

@LuisaLiu

This comment has been minimized.

Copy link
Author

@LuisaLiu LuisaLiu commented Mar 25, 2020

Hi, @rjust

Today, the initialization was successful. I regenerated the test case for the Mockito, then executed fix_test_suite.pl to remove the problematic test case, and then executed the run_mutation.pl script for mutation analysis.
Error information is as follows:

`liu@ubuntu:~/defects4j/framework/bin$ perl run_mutation.pl -p Mockito -d /home/liu/bishe_test/7-Randoop-regre-newversion/Mockito-test-fixd/Mockito/randoop/1 -o /home/liu/bishe_test/7-Randoop-regre-newversion/Mockito-test-fixd/Mockito/randoop/1-mutation
Executing test suite: Mockito-5f-randoop.1.tar.bz2
Extract test suite......................................................... OK
Checking out 42a24dde to /tmp/run_mutation.pl_3210_1585099980/5f........... OK
Init local repository...................................................... OK
Tag post-fix revision...................................................... OK
Run post-checkout hook..................................................... OK
Excluding broken/flaky tests............................................... OK
Excluding broken/flaky tests............................................... OK
Excluding broken/flaky tests............................................... OK
Initialize fixed program version........................................... OK
Apply patch................................................................ OK
Initialize buggy program version........................................... OK
Diff 42a24dde:fff76563..................................................... OK
Apply patch................................................................ OK
Tag pre-fix revision....................................................... OK
Check out program version: Mockito-5f...................................... OK
Compiling mutant definition (mml).......................................... OK
Running ant (mutate)....................................................... FAIL
Executed command: cd /tmp/run_mutation.pl_3210_1585099980/5f && ant -f /home/liu/defects4j/framework/projects/defects4j.build.xml -Dd4j.home=/home/liu/defects4j -Dd4j.dir.projects=/home/liu/defects4j/framework/projects -Dbasedir=/tmp/run_mutation.pl_3210_1585099980/5f -Dbuild.compiler=major.ant.MajorCompiler mutate 2>&1
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1G; support was removed in 8.0
Buildfile: /home/liu/defects4j/framework/projects/defects4j.build.xml

check.classes.uptodate:

mutate:

compile:

gradle.compile:

ant.compile:

gradle.compile.mutants:
[exec] Downloading file:/home/liu/defects4j/framework/lib/build_systems/gradle/dists/gradle-2.2.1-all.zip
[exec] ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
[exec] Unzipping /tmp/run_mutation.pl_3210_1585099980/5f/.gradle_local_home/wrapper/dists/gradle-2.2.1-all/7fs7sr9p2qcjed17f6spvl5591/gradle-2.2.1-all.zip to /tmp/run_mutation.pl_3210_1585099980/5f/.gradle_local_home/wrapper/dists/gradle-2.2.1-all/7fs7sr9p2qcjed17f6spvl5591
[exec] Set executable permissions for: /tmp/run_mutation.pl_3210_1585099980/5f/.gradle_local_home/wrapper/dists/gradle-2.2.1-all/7fs7sr9p2qcjed17f6spvl5591/gradle-2.2.1/bin/gradle
[exec] Parallel execution is an incubating feature.
[exec] :buildSrc:clean UP-TO-DATE
[exec] :buildSrc:compileJava UP-TO-DATE
[exec] Note: Some input files use unchecked or unsafe operations.
[exec] Note: Recompile with -Xlint:unchecked for details.
[exec] :buildSrc:compileGroovy
[exec] :buildSrc:processResources
[exec] :buildSrc:classes
[exec] :buildSrc:jar
[exec] :buildSrc:assemble
[exec] :buildSrc:compileTestJava UP-TO-DATE
[exec] :buildSrc:compileTestGroovy
[exec] :buildSrc:processTestResources UP-TO-DATE
[exec] :buildSrc:testClasses
[exec] :buildSrc:test
[exec] :buildSrc:check
[exec] :buildSrc:build
[exec] Download https://jcenter.bintray.com/com/android/tools/build/gradle/0.11.2/gradle-0.11.2.pom
[exec] POM relocation to an other version number is not fully supported in Gradle : xml-apis#xml-apis;2.0.2 relocated to xml-apis#xml-apis;1.0.b2.
[exec] Please update your dependency to directly use the correct version 'xml-apis#xml-apis;1.0.b2'.
[exec] Resolution will only pick dependencies of the relocated element. Artifacts and other metadata will be ignored.
[exec] Version: 2.0.3-beta
[exec] :compileJava
[exec] Major: Mutation enabled using mml file: /tmp/run_mutation.pl_3210_1585099980/5f/.mml/default.mml.bin
[exec] Note: Some input files use or override a deprecated API.
[exec] Note: Recompile with -Xlint:deprecation for details.
[exec] Note: Some input files use unchecked or unsafe operations.
[exec] Note: Recompile with -Xlint:unchecked for details.
[exec] #Generated Mutants: 16 (1820 ms)
[exec] :processResources UP-TO-DATE
[exec] :classes
[exec] :extTest:compileJava UP-TO-DATE
[exec] :extTest:processResources UP-TO-DATE
[exec] :extTest:classes UP-TO-DATE
[exec] Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1G; support was removed in 8.0
[exec] :jar
[exec] Buildfile: /tmp/run_mutation.pl_3210_1585099980/5f/build-ant.xml
[exec]
[exec] bndCore:
[exec] [bnd] Errors
[exec] [bnd] Unresolved references to [major.mutation] by class(es) on the Bundle-Classpath[Jar:dot]: [org/mockito/internal/verification/VerificationOverTimeImpl.class]
[exec] [bnd] /tmp/run_mutation.pl_3210_1585099980/5f/conf/mockito-core.bnd: bnd failed
[exec] [bnd] at aQute.bnd.ant.BndTask.executeBackwardCompatible(BndTask.java:128)
[exec]
[exec] BUILD FAILED
[exec] /tmp/run_mutation.pl_3210_1585099980/5f/build-ant.xml:25: Failed to build jar file:
[exec]
[exec] Total time: 0 seconds
[exec]
[exec] FAILURE: Build failed with an exception.
[exec]
[exec] * Where:
[exec] Build file '/tmp/run_mutation.pl_3210_1585099980/5f/build.gradle' line: 111
[exec]
[exec] * What went wrong:
[exec] Execution failed for task ':jar'.
[exec] > Process 'command 'ant'' finished with non-zero exit value 1
[exec]
[exec] * Try:
[exec] Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
[exec] [bnd] at aQute.bnd.ant.BndTask.execute(BndTask.java:28)
[exec] [bnd] at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
[exec] [bnd] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[exec] [bnd] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[exec] [bnd] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[exec] [bnd] at java.lang.reflect.Method.invoke(Method.java:497)
[exec] [bnd] at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
[exec] [bnd] at org.apache.tools.ant.Task.perform(Task.java:348)
[exec] [bnd] at org.apache.tools.ant.Target.execute(Target.java:392)
[exec] [bnd] at org.apache.tools.ant.Target.performTasks(Target.java:413)
[exec] [bnd] at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
[exec] [bnd] at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
[exec] [bnd] at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
[exec] [bnd] at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
[exec] [bnd] at org.apache.tools.ant.Main.runBuild(Main.java:811)
[exec] [bnd] at org.apache.tools.ant.Main.startAnt(Main.java:217)
[exec] [bnd] at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
[exec] [bnd] at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
[exec] :jar FAILED
[exec]
[exec] BUILD FAILED
[exec]
[exec] Total time: 1 mins 25.803 secs
[move] Moving 1 file to /tmp/run_mutation.pl_3210_1585099980/5f

BUILD FAILED
/home/liu/defects4j/framework/projects/defects4j.build.xml:178: The following error occurred while executing this line:
/home/liu/defects4j/framework/projects/Mockito/Mockito.build.xml:147: The following error occurred while executing this line:
/home/liu/defects4j/framework/projects/Mockito/Mockito.build.xml:113: /home/liu/defects4j/framework/projects/Mockito/Mockito.build.xml:120: exec returned: 1

Total time: 1 minute 28 seconds
No mutants generated for 5f! at run_mutation.pl line 307.
'

This is the test case I generated using Randoop:
Mockito-5f.zip

Thanks!

@rjust

This comment has been minimized.

Copy link
Owner

@rjust rjust commented Mar 25, 2020

@LuisaLiu,

Just to clarify, does the mutation analysis work for the generated test suites that you previously shared (Mockito-12 and Mockito-14)?

Also, did the mutation analysis work for Mockito-5 prior to the recent changes in Defects4J?

I'll look into the Mockito-5 issue.

Thanks,
René

@LuisaLiu

This comment has been minimized.

Copy link
Author

@LuisaLiu LuisaLiu commented Mar 26, 2020

Hi, @rjust

I used the latest version of defects4j to generate tests for all versions of the Mockito project, and then did mutation analysis. The mutation analysis of mockito-12 and mockito-14 was successful, but the following version failed:
1-10、18、20、21、35

This is the test case I generated using Randoop:
no-mutation.zip

And I have a question, why does the Major generate so few mutants for some versions of the Mockito project?

Thanks!

@rjust

This comment has been minimized.

Copy link
Owner

@rjust rjust commented Mar 26, 2020

Thanks @LuisaLiu, this is super useful information!

By default, Defects4J only mutates the class(es) modified by the bug fix. Presumably, the Mockito versions with very few mutants are the ones where the modified class is fairly small.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.