Test inclusion patterns are derived from application code

[nrainer]

Characteristics

• Issue Type: bug
• Reproducibility: always
• Severity: major
• Execution Environment: Ubuntu x64

The problem is that test inclusion patterns are derived from the application code. (As a consequence, no tests from projects without application code are executed.)

Old ticket: Tests from modules without application code are skipped

Description

Modules that contain exclusively tests (and no source code) are skipped from the analysis. For example, some projects have a dedicated module for integration tests, which is skipped in this case. (Actually, the tests of this project get skipped, but the mutations analysis is conducted. A result file is created.) Hence, not all tests of a project will get executed.

Steps to reproduce

1. Unzip the attached sample project. It is a multi-module project consisting of the modules core, extension, and tests. All modules contain tests, all modules except tests contain application code. The module tests does not contain application code and has no src/main/java folder.
2. After building the project with mvn clean install, run pit-mp using mvn eu.stamp-project:pitmp-maven-plugin:run
3. MyTest from the module tests covers a class from the module extension but does not get executed. See the attached mutation matrix.

Files

• sample-project
• mutation matrix

[nrainer]

Is the problem probably caused by mpMojo#shouldRun in line 214? (Line number of revision 72ebb6e)

[nrainer]

See pull request #26

[Cael35]

Thanks for the test case and the PR @nrainer
Sorry for the delay, I'm trying to catch up... :-)
I hope I can release a new version by next Friday.

[nrainer]

@CaelInria You are welcome :)
I think this issue can be closed since the corresponding PR has been merged.

[nrainer]

@CaelInria I fear that this problem was reintroduced with 1.3.5. :-/

The point is that we need to explicitly set the patterns for the test cases because otherwise PIT will derive them from the application code (src/main/java) (see [1]). As a consequence, the patterns created by PIT will only match tests in packages that also exist in the application code. Hence, tests in packages that do not exist in the application code will not be considered. A special case are modules without application code, in which PIT will include no test (because no packages in the application code exist).
Therefore, updateTests in PmpMojo [2] is definitively needed in my opinion. :)

[1] org.pitest.mutationtest.config.ReportOptions.getTargetTestsFilter():300
[2] https://github.com/STAMP-project/pitmp-maven-plugin/blob/master/src/main/java/org/pitest/maven/PmpMojo.java#L340

[Cael35]

@nrainer
Sorry, I probably miss something, but I tried on both dnoo and your example project-with-module-with-only-tests and I have a report for dnooIntegration (has only tests and no sources):

and for tests:

Is it the result your are expecting ?

[nrainer]

I fear that it is possible to answer the question based on the HTML report. From what I understand, this report lists the code under test (i. e., application code) but provides no direct information about the executed tests.

However, from what I remember when I was debugging PIT to analyze this ticket, I am quite sure that the change is necessary...

Can you please check if the tests located in dnooIntegration itself were executed (e.g., by using XML as output and finding a test there in killingTests)? (Note that when processing dnooIntegration tests from other modules might get executed.)
If so, does the package of these integration tests exist in one of the source folders of any module? If so, can you please rename the test package to a name that does not exist anywhere (and run mvn install afterwards) and check again if the tests were executed? I assume they won't be executed.

Let me know if you need further information. :)

[nrainer]

Probably easier:

• Create a multi-module-project with two modules A and B.
• Module A: add a simple class with one method in package org.pit.x.
• let module B depend on module A
• Module B: add a test case that tests the class from module A (and can kill the mutations) and place it in the test source folder in package org.pit.y.
• run the mutation analysis, check if the test is executed / if the mutations are killed. I think that the test won't be executed. --> NOT OK
• rename the package in module B to org.pit.x and rerun mvn install
• run the mutation analysis, check if the test is executed / if the mutations are killed. I think that the test will be executed. --> OK

[Cael35]

I checked the values... this issue is corrected in v1.3.6 :-)

[nrainer]

Cool, thanks for testing and adjusting! 👍