Allure in POM

Allure is a test report tool that creates a nifty HTML report from test execution results. Allure provides a Java API to define steps to structure your report and document the test execution. Additionally, attachments like screenshots and the page source can be added to the report. See the Allure documentation for more details.

Check out their demo report

How to configure Allure

The Allure plugin could be configured in several ways. Please see their documentation for more details. In the following example we use Maven to define configuration for version (2.5.0) of the resulting report template as well as the output folder for Allure's result files.

<build>
    <plugins>
        <plugin>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-maven</artifactId>
            <version>2.12.0</version>
            <configuration>
                <reportVersion>2.27.0</reportVersion>
                <resultsDirectory>${project.build.directory}/allure-results</resultsDirectory>
            </configuration>
        </plugin>   
    </plugins>
</build>

Since Allure writes their own test results besides once from JUnit the tool needs to be integrated into test cycle. Maven Surefire is a Maven plugin which covers test phase in lifecycle. You also need to add the following configuration to your pom.xml in order to get Allure reports working.

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>17</maven.compiler.source>
    <maven.compiler.target>17</maven.compiler.target>
    <surefire.version>3.2.5</surefire.version>
    <aspectj.version>1.9.21</aspectj.version>
    <allure.version>2.12.0</allure.version>
</properties>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>${surefire.version}</version>
            <configuration>
                <forkCount>4</forkCount><!-- parallel test execution -->
              <testFailureIgnore>true</testFailureIgnore>
                <argLine>-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"</argLine>
          <systemPropertyVariables>
            <allure.results.directory>${project.build.directory}/allure-results</allure.results.directory>
            <selenide.reports>${project.build.directory}/selenide-results</selenide.reports>
          </systemPropertyVariables>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.aspectj</groupId>
                    <artifactId>aspectjweaver</artifactId>
                    <version>${aspectj.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

How to generate reports

To create a report run the Maven goal mvn allure:report. That command will scan the test execution results and create the HTML report but it doesn't execute the tests. To execute tests and create report you should use mvn clean test allure:report that will clean the project execute tests and finally generate the report from test results.

Steps

There are different ways to define steps of your test case that are later on visible in the report. You can give functions static annotation

@Step("FirstStep")
public void myFirstStep()
{
    // do stuff
}

and also let passed variables be displayed in the report

@NeodymiumTest
public void myTest()
{
    step("first step");
    // do stuff
    
    step("second step");
    // do stuff
}

@Step("\"{stepName}\" step")
public void step(String stepName)
{
}

Allure supports steps are used to structure the test case. Steps are basically annotated functions but they appear in the report and their runtime will be measured. So you can see which step took what time to finish. See the following example.

public class SampleTest
{
    @NeodymiumTest
    public void testMethod()
    {
        step1();
        step2();
    }

    @Step
    public void step1()
    {
        step2();
    }

    @Step
    public void step2()
    {
        step3();
    }

    @Step
    public void step3()
    {
    }
}

This code will be displayed in the report as in the following screenshot

More details about steps and other see the official Allure documentation about Steps

Describing your test cases with annotations

Allure provides several annotations that help you to enhance your test report. The following annotations need to be applied to your test case.

@Owner("Lisa Smith")
@Severity(SeverityLevel.BLOCKER)
@Tag("smoke")
@Tag("registered")
public class RegisteredOrderTest extends AbstractTest
{
}

Furthermore, you can add general, issue and test management system links that help you to match other places of information with your test report.

@TmsLink("end-of-script-developer-and-now")
@Issue("148")
@Link(url = "https://ask.xceptance.de/t/end-of-script-developer-and-now/148", type = "custom", name = "DemoLink")
public class HomePageTest extends AbstractTest
{
}

To use the TmsLink and Issue annotations you need to specify the urls of the systems in your pom.xml as below.

<systemPropertyVariables>
    <allure.results.directory>${project.build.directory}/allure-results</allure.results.directory>
    <allure.link.issue.pattern>https://ask.xceptance.de/t/{}</allure.link.issue.pattern>
    <allure.link.tms.pattern>https://ask.xceptance.de/t/{}</allure.link.tms.pattern>
    <selenide.reports>${project.build.directory}/selenide-results</selenide.reports>
</systemPropertyVariables>

History

Allure reports can have a trend graph that indicates test execution results over the past iterations.

In order to get the graph created during allure:report you need to provide the result (history) files of the previous test executions. To accomplish that in an easy way we provide a script in example and template project that does the necessary file copy actions. For Linux/MacOS you need to run testAndBuildReport.sh. For Microsoft Windows os you need to run testAndBuildReport.bat. These scripts copy the required files to their destination and run maven afterwards (mvn test allure:report to be precise)