Skip to content
Gradle plugin to retry tests that have failed to mitigate test flakiness.
Java Groovy Kotlin
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/workflows Official Gradle Wrapper Validation GitHub Action Jan 16, 2020
.idea Apply code style, and configure IDEA build for delegation to Gradle Dec 16, 2019
.teamcity Fix developer snapshot teamcity job invoking the right gradle task Jan 20, 2020
buildSrc/src/main/groovy/org/gradle/testretry/build Handle versions in major.minor form Dec 16, 2019
config
docs/images Add dropshadow to images Dec 20, 2019
gradle
plugin cleanup and improved readability Jan 23, 2020
sample-tests
samples/sample-junit5 Update samples to plugin version 1.0.0 Dec 17, 2019
.gitignore Apply code style, and configure IDEA build for delegation to Gradle Dec 16, 2019
LICENSE Update README Dec 11, 2019
README.adoc Polish docs Dec 20, 2019
build.gradle.kts Add check for java8 in use in releaseCheck Dec 16, 2019
gradlew Initial import Nov 27, 2019
gradlew.bat Initial import Nov 27, 2019
settings.gradle.kts Enable build cache Dec 16, 2019

README.adoc

Test Retry Gradle Plugin

A Gradle plugin that augments Gradle’s built-in test task with the ability to retry tests that have failed.

Version GitHub license

What it does

The plugin causes failed tests to be retried within the same task. After executing all tests, any failed tests are retried. The process repeats with tests that continue to fail until the maximum specified number of retries has been attempted, or there are no more failing tests.

By default, all failed tests passing on retry prevents the test task from failing. This mode prevents flaky tests from causing build failure. This setting can be changed so that flaky tests cause build failure, which can be used to identify flaky tests.

When something goes badly wrong and all tests start failing, it can be preferable to not keep retrying tests. This can happen for example if a disk fills up or a required database is not available. To avoid this, the plugin can be configured to stop retrying after a certain number of total test failures.

Usage

The plugin is available from the Gradle plugin portal with ID org.gradle.test-retry. It is compatible with Gradle 5.0 and later versions.

By default, retrying is not enabled. Retrying is configured per test task via the retry extension added to each task by the plugin.

build.gradle:
plugins {
    id 'org.gradle.test-retry' version '1.0.0'
}

test {
    retry {
        maxRetries = 3
        maxFailures = 20
        failOnPassedAfterRetry = true
    }
}

The retry extension is of the following type:

package org.gradle.testretry;

import org.gradle.api.provider.Property;
import org.gradle.api.tasks.testing.Test;

/**
 * Allows configuring test retry mechanics.
 * <p>
 * This extension is added with the name 'retry' to all {@link Test} tasks.
 */
public interface TestRetryTaskExtension {

    /**
     * Whether tests that initially fail and then pass on retry should fail the task.
     * <p>
     * This setting defaults to {@code false},
     * which results in the task not failing if all tests pass on retry.
     * <p>
     * This setting has no effect if {@link Test#getIgnoreFailures()} is set to true.
     *
     * @return whether tests that initially fail and then pass on retry should fail the task
     */
    Property<Boolean> getFailOnPassedAfterRetry();

    /**
     * The maximum number of times to retry an individual test.
     * <p>
     * This setting defaults to {@code 0}, which results in no retries.
     * Any value less than 1 disables retrying.
     *
     * @return the maximum number of times to retry an individual test
     */
    Property<Integer> getMaxRetries();

    /**
     * The maximum number of test failures that are allowed before retrying is disabled.
     * <p>
     * The count applies to each round of test execution.
     * For example, if maxFailures is 5 and 4 tests initially fail and then 3 again on retry,
     * this will not be considered too many failures and retrying will continue (if maxRetries {@literal >} 1).
     * If 5 or more tests were to fail initially then no retry would be attempted.
     * <p>
     * This setting defaults to {@code 0}, which results in no limit.
     * Any value less than 1 results in no limit.
     *
     * @return the maximum number of test failures that are allowed before retrying is disabled
     */
    Property<Integer> getMaxFailures();

}

Supported test frameworks

Other versions are likely to work as well, but are not tested.

Framework Version Tested

JUnit4

4.12

JUnit5

5.5.2

Spock

1.3-groovy-2.5

TestNG

7.0.0

Parameterized tests

In a few cases, test selection for testing frameworks limits the granularity at which tests can be retried. In each case, this plugin retries at worst at method level. For JUnit5 @ParameterizedTest, TestNG @Test(dataProvider = "…​"), and Spock @Unroll tests the plugin will retry the entire method with all parameters including those that initially passed.

Test dependencies

The plugin supports retrying Spock @Stepwise tests and TestNG @Test(dependsOn = { … }) tests.

  • Upstream tests (those that the failed test depends on) are run because a flaky test may depend on state from the prior execution of an upstream test.

  • Downstream tests are run because a flaky test causes any downstream tests to be skipped in the initial test run.

Reporting

Gradle

Each execution of a test is discretely reported in Gradle-generated XML and HTML reports.

Gradle test reporting

Gradle flaky test reporting

Similar to the XML and HTML reports, the console log will also report each individual test execution. Before retrying a failed test, Gradle will execute the whole test suite of the test task. This means that all executions of the same test may not be grouped in the console log.

Gradle console reporting

Gradle Enterprise

Gradle build scans (--scan option) report discrete test executions as "Execution [N of total]" and will mark a test with both a failed and then a passed outcome as flaky.

Gradle build scan reporting

Flaky tests can also be visualized across many builds using the Gradle Enterprise Tests Dashboard.

Gradle Enterprise top tests report

IDEs

The plugin has been tested with IDEA, Eclipse IDE and Netbeans.

IDEA

When delegating test execution to Gradle, each execution is reported discretely as for the test reports. Running tests without Gradle delegation causes tests to not be retried.

IDEA test reporting

Eclipse

When delegating test execution to Gradle, each execution is reported discretely as for the test reports. Running tests without Gradle delegation causes tests to not be retried.

Eclipse test reporting

Netbeans

Netbeans only shows the last execution of a test.

Netbeans test reporting

CI tools

The plugin has been tested with the reporting of TeamCity and Jenkins.

TeamCity

Flaky tests (tests being executed multiple times but with different results) are detected by TeamCity and marked as flaky. TeamCity lists each test that was executed and how often it was run in the build.

Teamcity test reporting

Jenkins

Jenkins reports each test execution discretely.

Jenkins test reporting

You can’t perform that action at this time.