jQAssistant plugin for analyzing impact on tests based on changes in Git repositories.
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
src
.editorconfig
.gitignore
LICENSE
README.adoc
pom.xml

README.adoc

jQAssistant Test Impact Analysis Plugin

The project provides a jQAssistant plugin for generating test suites based on changes in the version control system (i.e. Git). This allows to selectively run expensive tests to provide fast feedback. Furthermore an experimental feature identifies potential test gap, i.e. changed or added Java classes without unit tests.

How It Works

The test impact analysis covers the following steps:

  1. Determine source files that have been changed in the Git repository by evaluating

    • the last commit in the current branch or

    • all commits in the current branch that are not contained in a given base (i.e. reference) branch

  2. Match the changed source files to their corresponding Java types (classes, interfaces, enumerations or annotations)

  3. Find JUnit test classes that have a dependency to these types and write their file names to test suite files.

The plugin provides concepts (i.e. queries) and a report implementation for jQAssistant to perform these steps.

The test suites are represented by files which can be passed as parameters to Maven Surefire Plugin or Maven Failsafe Plugin via the includesFile option.

Prerequisites

  • Java 7 or higher

  • Maven 3.2.5 or higher

  • jQAssistant 1.3.0 or higher (see below)

Setup

The plugin can be enabled in a Maven based project by adding it as a dependency to the jQAssistant Maven plugin:

pom.xml
<build>
    <plugins>
        <plugin>
            <groupId>com.buschmais.jqassistant</groupId>
            <artifactId>jqassistant-maven-plugin</artifactId>
            <version>1.4.0</version>
            <executions>
                <execution>
                    <id>default-cli</id>
                    <goals>
                        <goal>scan</goal>
                        <goal>analyze</goal>
                    </goals>
                    <configuration>
                        <failOnSeverity>MAJOR</failOnSeverity>
                        <warnOnSeverity>MINOR</warnOnSeverity>
                        <useExecutionRootAsProjectRoot>true</useExecutionRootAsProjectRoot>
                        <scanIncludes>
                            <scanInclude>
                                <path>${project.basedir}/.git</path>
                            </scanInclude>
                        </scanIncludes>
                        <reportProperties>                                               (1)
                            <testImpactAnalysis.report.directory>${session.executionRootDirectory}/target/testimpactanalysis</testImpactAnalysis.report.directory>
                        </reportProperties>
                    </configuration>
                </execution>
            </executions>
            <dependencies>
                <dependency>                                                             (2)
                    <groupId>org.jqassistant.contrib.plugin</groupId>
                    <artifactId>jqassistant-test-impact-analysis-plugin</artifactId>
                    <version>1.4.0</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>
  1. Configures the directory where the jQAssistant test impact analysis plugin will create the files (one per artifact)

  2. Declares the plugin as dependency for jQAssistant

Furthermore profiles should be defined for making execution easier:

pom.xml
<profiles>
    <profile>
        <id>create-commit-testsuite</id>                                                                     (1)
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>com.buschmais.jqassistant</groupId>
                    <artifactId>jqassistant-maven-plugin</artifactId>
                    <configuration>
                        <concepts>
                            <concept>test-impact-analysis:SurefireSuiteForLastGitCommit</concept>            (2)
                        </concepts>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

    <profile>
        <id>create-branch-testsuite</id>                                                                     (3)
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>com.buschmais.jqassistant</groupId>
                    <artifactId>jqassistant-maven-plugin</artifactId>
                    <configuration>
                        <concepts>
                            <concept>test-impact-analysis:SurefireSuiteForCurrentBranch</concept>            (4)
                        </concepts>
                        <ruleParameters>
                            <testImpactAnalysisGitBaseBranch>heads/master</testImpactAnalysisGitBaseBranch>  (5)
                        </ruleParameters>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

    <profile>
        <id>run-testsuite</id>                                                                               (6)
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <includesFile>${session.executionRootDirectory}/target/testimpactanalysis/${project.artifactId}</includesFile>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>com.buschmais.jqassistant</groupId>
                    <artifactId>jqassistant-maven-plugin</artifactId>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profile>
  1. Defines the profile to be used for creating a test suite based on the last Git commit

  2. Activates the concept test-impact-analysis:SurefireSuiteForLastGitCommit

  3. Defines the profile to be used for creating a test suite based on the commits within the current Git branch

  4. Activates the concept test-impact-analysis:SurefireSuiteForCurrentBranch

  5. Defines the base branch to use for determining changes in the current branch (optional, default: "heads/master")

  6. Defines the profile for running the tests defined in the generated test suite

Creating And Running A Test Suite

The profiles create-commit-testsuite and create-branch-testsuite are used to create the required test suite files. Either one of the following commands needs to be executed:

mvn clean verify -Pcreate-commit-testsuite
mvn clean verify -Pcreate-branch-testsuite

The test suite files (one per artifact) are available in the 'target/testimpactanalysis' folder of the module where Maven has been executed, e.g.

target/testimpactanalysis/spring-petclinic
org/springframework/samples/petclinic/web/OwnerControllerTests.java
org/springframework/samples/petclinic/web/PetControllerTests.java
org/springframework/samples/petclinic/web/PetTypeFormatterTests.java
org/springframework/samples/petclinic/web/VetControllerTests.java
org/springframework/samples/petclinic/web/VisitControllerTests.java

The following command triggers a build which only executes the tests which are defined in the generated test suite files:

mvn verify -Prun-testsuite

Selecting Specific Tests

It is possible to select only specific tests to be included in the test suite files. Therefore specific rules can be created using Cypher (Neo4j query language) which perform required filtering. The rules must be located in a file that can be interpreted by jQAssistant. Allowed formats are Asciidoc or XML, the files must be located in the folder "jqassistant" in the root module of the Maven project:

Example project structure
my-project/
          /jqassistant/my-rules.adoc
          /module1/pom.xml
          /module2/pom.xml
          /pom.xml

Integration Tests

A case might be that only integration tests shall be selected for execution by the the Maven Failsafe Plugin. The following two rules select test classes by checking their name for the suffix "IT":

jqassistant/my-rules.adoc
= My Rules

[[my-rules:IntegrationTestsForLastGitCommit]]                                       (1)
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TestsAffectedByLastGitCommit",reportType="surefire-suite"]
.Reports all integration test classes affected by the last Git commit.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)
OPTIONAL MATCH
  (artifact)-[:CONTAINS]->(test:Affected:Test)
WHERE
  test.name ends with "IT"                                                          (2)
RETURN
  artifact as Artifact, collect(test) as Tests
----

[[my-rules:IntegrationTestsForCurrentGitBranch]]                                    (3)
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TestsAffectedByCurrentGitBranch",reportType="surefire-suite"]
.Reports all integration test classes affected by commits in the current Git branch.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)
OPTIONAL MATCH
  (artifact)-[:CONTAINS]->(test:Affected:Test)
WHERE
  test.name ends with "IT"
RETURN
  artifact as Artifact, collect(test) as Tests
----
  1. Declares the concept for integration tests affected by the last Git commit

  2. The filter clause for test classes having a name with the suffix "IT"

  3. Declares the concept for integration tests affected by commits within the current Git branch

The result of the concepts are reported as "surefire-suite". This report type requires each row to provide two columns:

Artifact

The artifact where affected test classes are located.

Tests

The collection of affected tests for the artifact.

For activating the rules the above described Maven profiles need to be adopted accordingly.

Impact On Artifacts

For projects consisting of multiple Maven modules it might be interesting to select all tests in Maven modules that depend on modules where Java types have been changed. The according rules would be as follows:

jqassistant/my-rules.adoc
= My Rules

[[my-rules:TestsInAffectedArtifactsForLastGitCommit]]
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TypesChangedByLastGitCommit,junit4:TestClass",reportType="surefire-suite"]
.Reports all test classes of artifacts affected by the last Git commit.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)-[:CONTAINS]->(:Type:Changed),
  (:Maven:Project)-[:CREATES]->(affectedArtifact:Artifact),
  shortestPath((affectedArtifact)-[:DEPENDS_ON*0..]->(artifact))
WITH DISTINCT
  affectedArtifact
MATCH
  (affectedArtifact)-[:CONTAINS]->(test:Type:Test)
RETURN
  affectedArtifact as Artifact, collect(test) as Tests
----

[[my-rules:TestsInAffectedArtifactsForCurrentGitBranch]]
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TypesChangedByCurrentGitBranch,junit4:TestClass",reportType="surefire-suite"]
.Reports all test classes of artifacts affected by commits in the current Git branch.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)-[:CONTAINS]->(:Type:Changed),
  (:Maven:Project)-[:CREATES]->(affectedArtifact:Artifact),
  shortestPath((affectedArtifact)-[:DEPENDS_ON*0..]->(artifact))
WITH DISTINCT
  affectedArtifact
MATCH
  (affectedArtifact)-[:CONTAINS]->(test:Type:Test)
RETURN
  affectedArtifact as Artifact, collect(test) as Tests
----

Test Gap

Note
This feature is considered experimental, Feedback is highly appreciated.

The plugin provides two additional constraints for test gap analysis. Both determine public methods of changed Java types that are not invoked by JUnit test methods:

test-impact-analysis:TestGapForLastGitCommit

Determines changes from the last Git commit

test-impact-analysis:TestGapForCurrentGitBranch

Determines changes within the current Git branch

The constraints may be verified from the command line:

mvn verify -Djqassistant.constraints=test-impact-analysis:TestGapForLastGitCommit
mvn verify -Djqassistant.constraints=test-impact-analysis:TestGapForCurrentGitBranch

If jQAssistant is already used in the Maven project it is recommended to include the desired constraint in a group that is executed:

jqassistant/my-rules.adoc
= My Rules

[[default]]
[role=group,includesConstraints="test-impact-analysis:TestGapForCurrentGitBranch"]
== Default Rules

Configuration

The Surefire Report plugin accepts several options that might be passed as reportProperties in the configuration section of the jQAssistant Maven plugin:

Property Description Default

testImpactAnalysis.report.directory

Specifies the directory where the test suite files will be written

jqassistant/report

testImpactAnalysis.surefire.file

If provided all affected test names will be written to one file with that name (relative to the directory)

testImpactAnalysis.surefire.artifactColumn

The name of the column providing the artifact containing a test

Artifact

testImpactAnalysis.surefire.testsColumn

The name of the column providing the collection of tests for a specific artifact

Tests

Note
The properties specifying the artifactColumn and testColumn must reflect the columns used in the RETURN clause of the query, e.g. by default 'RETURN artifact as Artifact, collect(test) as Tests'.

Limitations

At the moment only test classes are detected having a direct dependency to

  • a changed type

  • any super type of a changed type (e.g. interfaces, super classes, etc.)

  • any derived class of a changed type

Evaluation of transitive relations is considered for a future release.

Known Problems

  1. The following error may occur while creating a branch test suite:

Analysis failed.: Cannot execute query for rule 'Concept{id='test-impact-analysis:ChangeCommitsInCurrentGitBranch', description='Adds a label 'Change' to all commits in the current Git branch which are not included in the base branch.', ruleSource=META-INF/jqassistant-rules/test-impact-analysis.xml}'. All parts of the pattern must either directly or indirectly be connected to at least one bound entity. These identifiers were found to be disconnected:   UNNAMED322, baseBranch, baseHead -> [Help 1]

The problem is caused by the shortestPath function in Neo4j v2 and can be solved by switching jQAssistant to use Neo4j v3:

mvn clean verify -Pcreate-branch-testsuite -Djqassistant.neo4jVersion=3

For convenience this setting can be stored in a file in the top level module of the Maven project:

.mvn/maven.config
-Djqassistant.neo4jVersion=3

Feedback

Please report any issues here.

Acknowledgements

The plugin could not provide its functionality without the support of the following open source projects: