Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
src
 
 
 
 
 
 
 
 
 
 

README.adoc

jQAssistant Test Impact Analysis Plugin

The project provides a jQAssistant plugin for generating test suites based on changes in the version control system (i.e. Git). This allows to selectively run expensive tests to provide fast feedback. Furthermore an experimental feature identifies potential test gap, i.e. changed or added Java classes without unit tests.

How It Works

The test impact analysis covers the following steps:

  1. Determine source files that have been changed in the Git repository by evaluating

    • the last commit in the current branch or

    • all commits in the current branch that are not contained in a given base (i.e. reference) branch

  2. Match the changed source files to their corresponding Java types (classes, interfaces, enumerations or annotations)

  3. Find JUnit test classes that have a dependency to these types and write their file names to test suite files.

The plugin provides concepts (i.e. queries) and a report implementation for jQAssistant to perform these steps.

The test suites are represented by files which can be passed as parameters to Maven Surefire Plugin or Maven Failsafe Plugin via the includesFile option.

Prerequisites

  • Java 8 or higher

  • Maven 3.2.5 or higher

  • jQAssistant 1.6.0 or higher (see below)

Setup

The plugin can be enabled in a Maven based project by adding it as a dependency to the jQAssistant Maven plugin:

pom.xml
<build>
    <plugins>
        <plugin>
            <groupId>com.buschmais.jqassistant</groupId>
            <artifactId>jqassistant-maven-plugin</artifactId>
            <version>1.6.0</version>
            <executions>
                <execution>
                    <id>default-cli</id>
                    <goals>
                        <goal>scan</goal>
                        <goal>analyze</goal>
                    </goals>
                    <configuration>
                        <failOnSeverity>MAJOR</failOnSeverity>
                        <warnOnSeverity>MINOR</warnOnSeverity>
                        <useExecutionRootAsProjectRoot>true</useExecutionRootAsProjectRoot>
                        <scanIncludes>
                            <scanInclude>
                                <path>${project.basedir}/.git</path>
                            </scanInclude>
                        </scanIncludes>
                        <reportProperties>                                               <!--(1)-->
                            <testImpactAnalysis.report.directory>${session.executionRootDirectory}/target/testimpactanalysis</testImpactAnalysis.report.directory>
                        </reportProperties>
                    </configuration>
                </execution>
            </executions>
            <dependencies>
                <dependency>                                                             <!--(2)-->
                    <groupId>org.jqassistant.contrib.plugin</groupId>
                    <artifactId>jqassistant-test-impact-analysis-plugin</artifactId>
                    <version>1.6.0</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>
  1. Configures the directory where the jQAssistant test impact analysis plugin will create the files (one per artifact)

  2. Declares the plugin as dependency for jQAssistant

Furthermore profiles should be defined for making execution easier:

pom.xml
<profiles>
    <profile>
        <id>create-commit-testsuite</id>                                                                     <!--(1)-->
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>com.buschmais.jqassistant</groupId>
                    <artifactId>jqassistant-maven-plugin</artifactId>
                    <configuration>
                        <concepts>
                            <concept>test-impact-analysis:SurefireSuiteForLastGitCommit</concept>            <!--(2)-->
                        </concepts>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

    <profile>
        <id>create-branch-testsuite</id>                                                                     <!--(3)-->
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>com.buschmais.jqassistant</groupId>
                    <artifactId>jqassistant-maven-plugin</artifactId>
                    <configuration>
                        <concepts>
                            <concept>test-impact-analysis:SurefireSuiteForCurrentBranch</concept>            <!--(4)-->
                        </concepts>
                        <ruleParameters>
                            <testImpactAnalysisGitBaseBranch>heads/master</testImpactAnalysisGitBaseBranch>  <!--(5)-->
                        </ruleParameters>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

    <profile>
        <id>run-testsuite</id>                                                                               <!--(6)-->
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <includesFile>${session.executionRootDirectory}/target/testimpactanalysis/${project.artifactId}</includesFile>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>com.buschmais.jqassistant</groupId>
                    <artifactId>jqassistant-maven-plugin</artifactId>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profile>
  1. Defines the profile to be used for creating a test suite based on the last Git commit

  2. Activates the concept test-impact-analysis:SurefireSuiteForLastGitCommit

  3. Defines the profile to be used for creating a test suite based on the commits within the current Git branch

  4. Activates the concept test-impact-analysis:SurefireSuiteForCurrentBranch

  5. Defines the base branch to use for determining changes in the current branch (optional, default: "heads/master")

  6. Defines the profile for running the tests defined in the generated test suite

Creating And Running A Test Suite

The profiles create-commit-testsuite and create-branch-testsuite are used to create the required test suite files. Either one of the following commands needs to be executed:

mvn clean verify -Pcreate-commit-testsuite
mvn clean verify -Pcreate-branch-testsuite

The test suite files (one per artifact) are available in the 'target/testimpactanalysis' folder of the module where Maven has been executed, e.g.

target/testimpactanalysis/spring-petclinic
org/springframework/samples/petclinic/web/OwnerControllerTests.java
org/springframework/samples/petclinic/web/PetControllerTests.java
org/springframework/samples/petclinic/web/PetTypeFormatterTests.java
org/springframework/samples/petclinic/web/VetControllerTests.java
org/springframework/samples/petclinic/web/VisitControllerTests.java

The following command triggers a build which only executes the tests which are defined in the generated test suite files:

mvn verify -Prun-testsuite

Selecting Specific Tests

It is possible to select only specific tests to be included in the test suite files. Therefore specific rules can be created using Cypher (Neo4j query language) which perform required filtering. The rules must be located in a file that can be interpreted by jQAssistant. Allowed formats are Asciidoc or XML, the files must be located in the folder "jqassistant" in the root module of the Maven project:

Example project structure
my-project/
          /jqassistant/my-rules.adoc
          /module1/pom.xml
          /module2/pom.xml
          /pom.xml

Integration Tests

A case might be that only integration tests shall be selected for execution by the the Maven Failsafe Plugin. The following two rules select test classes by checking their name for the suffix "IT":

jqassistant/my-rules.adoc
= My Rules

[[my-rules:IntegrationTestsForLastGitCommit]]                                       // (1)
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TestsAffectedByLastGitCommit",reportType="surefire-suite"]
.Reports all integration test classes affected by the last Git commit.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)
OPTIONAL MATCH
  (artifact)-[:CONTAINS]->(test:Affected:Test)
WHERE
  test.name ends with "IT"                                                          // (2)
RETURN
  artifact as Artifact, collect(test) as Tests
----

[[my-rules:IntegrationTestsForCurrentGitBranch]]                                    // (3)
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TestsAffectedByCurrentGitBranch",reportType="surefire-suite"]
.Reports all integration test classes affected by commits in the current Git branch.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)
OPTIONAL MATCH
  (artifact)-[:CONTAINS]->(test:Affected:Test)
WHERE
  test.name ends with "IT"
RETURN
  artifact as Artifact, collect(test) as Tests
----
  1. Declares the concept for integration tests affected by the last Git commit

  2. The filter clause for test classes having a name with the suffix "IT"

  3. Declares the concept for integration tests affected by commits within the current Git branch

The result of the concepts are reported as "surefire-suite". This report type requires each row to provide two columns:

Artifact

The artifact where affected test classes are located.

Tests

The collection of affected tests for the artifact.

For activating the rules the above described Maven profiles need to be adopted accordingly.

Impact On Artifacts

For projects consisting of multiple Maven modules it might be interesting to select all tests in Maven modules that depend on modules where Java types have been changed. The according rules would be as follows:

jqassistant/my-rules.adoc
= My Rules

[[my-rules:TestsInAffectedArtifactsForLastGitCommit]]
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TypesChangedByLastGitCommit,junit4:TestClass",reportType="surefire-suite"]
.Reports all test classes of artifacts affected by the last Git commit.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)-[:CONTAINS]->(:Type:Changed),
  (:Maven:Project)-[:CREATES]->(affectedArtifact:Artifact),
  shortestPath((affectedArtifact)-[:DEPENDS_ON*0..]->(artifact))
WITH DISTINCT
  affectedArtifact
MATCH
  (affectedArtifact)-[:CONTAINS]->(test:Type:Test)
RETURN
  affectedArtifact as Artifact, collect(test) as Tests
----

[[my-rules:TestsInAffectedArtifactsForCurrentGitBranch]]
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TypesChangedByCurrentGitBranch,junit4:TestClass",reportType="surefire-suite"]
.Reports all test classes of artifacts affected by commits in the current Git branch.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)-[:CONTAINS]->(:Type:Changed),
  (:Maven:Project)-[:CREATES]->(affectedArtifact:Artifact),
  shortestPath((affectedArtifact)-[:DEPENDS_ON*0..]->(artifact))
WITH DISTINCT
  affectedArtifact
MATCH
  (affectedArtifact)-[:CONTAINS]->(test:Type:Test)
RETURN
  affectedArtifact as Artifact, collect(test) as Tests
----

Test Gap

Note
This feature is considered experimental, Feedback is highly appreciated.

The plugin provides two additional constraints for test gap analysis. Both determine public methods of changed Java types that are not invoked by JUnit test methods:

test-impact-analysis:TestGapForLastGitCommit

Determines changes from the last Git commit

test-impact-analysis:TestGapForCurrentGitBranch

Determines changes within the current Git branch

The constraints may be verified from the command line:

mvn verify -Djqassistant.constraints=test-impact-analysis:TestGapForLastGitCommit
mvn verify -Djqassistant.constraints=test-impact-analysis:TestGapForCurrentGitBranch

If jQAssistant is already used in the Maven project it is recommended to include the desired constraint in a group that is executed:

jqassistant/my-rules.adoc
= My Rules

[[default]]
[role=group,includesConstraints="test-impact-analysis:TestGapForCurrentGitBranch"]
== Default Rules

Configuration

The Surefire Report plugin accepts several options that might be passed as reportProperties in the configuration section of the jQAssistant Maven plugin:

Property Description Default

testImpactAnalysis.report.directory

Specifies the directory where the test suite files will be written

jqassistant/report

testImpactAnalysis.surefire.file

If provided all affected test names will be written to one file with that name (relative to the directory)

testImpactAnalysis.surefire.artifactColumn

The name of the column providing the artifact containing a test

Artifact

testImpactAnalysis.surefire.testsColumn

The name of the column providing the collection of tests for a specific artifact

Tests

Note
The properties specifying the artifactColumn and testColumn must reflect the columns used in the RETURN clause of the query, e.g. by default 'RETURN artifact as Artifact, collect(test) as Tests'.

Feedback

Please report any issues here.

Acknowledgements

The plugin could not provide its functionality without the support of the following open source projects:

About

jQAssistant plugin for analyzing impact on tests based on changes in Git repositories.

Topics

Resources

License

Packages

No packages published

Languages

You can’t perform that action at this time.