Add a way to repeat the entire test suite

[Thihup]

Currently, JUnit has the option of using the @RepeatedTest annotation to allow a test to run multiple times.

Using the JITWatch, we can see how the application is behaving, so using the test suite would be a great way to get this information. However, running the tests one time each we cannot give the JIT enough time to really optimize.

In summary, to achieve this, now I'm replacing all @Test() with @RepeatedTest(4500) and running the test suit via console, like:

mvn package -DskipTests
java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:+LogCompilation -XX:LogFile=mylogfile.log  -jar ~/Downloads/junit-platform-console-standalone-1.8.0-M1.jar --class-path "..." --scan-classpath

This way, after running the test suite, I get mylogfile.log with a lot of JIT decisions. I'd like to know if it would be of use to have an option that could repeat all the tests instead of having to do a search and replace in the test suite.

(Background: junit-pioneer/junit-pioneer#472)

[marcphilipp]

I don't think this use case is general enough to warrant a new core feature. I'd recommend writing a small main method that uses the Launcher API directly with a for loop repeating the execution the desired number of times.

[justincranford]

Please reconsider adding a repeat annotation for @Suite.

I am looking for this exact feature, but to cover more use cases than just profiling. Here are a few example use cases:

1. Performance: CPU or Memory profiling using many @Suite repeats (consecutive or concurrent)
2. Robustness: Stress testing (e.g. DB deadlocks) using many concurrent repeats
3. Support: Reproduce customer issues related to 1 or 2
4. Analytics: Parse and analyze structured execution time data, to identify 95%/99% outliers in performance at a Suite, Class, or Method level
5. Automation: Create CI/CD alerts on data from 4 to notify when performance has regressed below some threshold

I tried the suggestion to create a small main that uses the Launcher. It works, but I don't get any reports of tests discovered, execution times, and success/error/failures. That also means no data for IDEs such as IntelliJ or Eclipse to report on, even though I am running the Launcher CLI via IDE.

Please reconsider adding a repeat annotation for @Suite.

[justincranford]

I am attaching my @Suite vs CLI Launcher example code below.

Notice the amount of custom code in Cli.Utils class to do custom printouts of discovered classes and methods, as well as custom reporting of execution times. The output is custom, not very user friendly, and not fine-grained.

I would love to see first class support for @Suite repeats in JUnit.

package com.github.justincranford.sweetrepeats;

import org.junit.platform.engine.DiscoverySelector;
import org.junit.platform.engine.discovery.ClassNameFilter;
import org.junit.platform.engine.discovery.DiscoverySelectors;
import org.junit.platform.launcher.Launcher;
import org.junit.platform.launcher.LauncherDiscoveryRequest;
import org.junit.platform.launcher.TestIdentifier;
import org.junit.platform.launcher.TestPlan;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;
import org.junit.platform.suite.api.IncludeClassNamePatterns;
import org.junit.platform.suite.api.SelectPackages;

import java.util.Arrays;
import java.util.stream.IntStream;

import static org.aspectj.util.LangUtil.isEmpty;
import static org.assertj.core.util.Arrays.isNullOrEmpty;

/**
 * JUnit Suites for local batch testing (e.g. non-repeats, can be used for functional testing)
 *     - com.github.justincranford.sweetrepeats.JUnit.Suite.All
 *     - com.github.justincranford.sweetrepeats.JUnit.Suite.Test
 *     - com.github.justincranford.sweetrepeats.JUnit.Suite.IT
 * JUnit CLI runners for local batch testing (e.g. repeats supported, can be used for local performance dev testing)
 *     - com.github.justincranford.sweetrepeats.JUnit.Cli.All.main     2 10
 *     - com.github.justincranford.sweetrepeats.JUnit.Cli.Package.main com.github.justincranford.sweetrepeats.util     2 20
 *     - com.github.justincranford.sweetrepeats.JUnit.Cli.Class.main   com.github.justincranford.sweetrepeats.util.Base64UtilsTest     2 50
 *     - com.github.justincranford.sweetrepeats.JUnit.Cli.Method.main  com.github.justincranford.sweetrepeats.util.Base64UtilsTest$NestedBase64UrlTest#generateKeyUrl32     2 100
 */
public class JUnit {
    private static final String DEFAULT_PACKAGE = "com.github.justincranford";
    private static final String TEST_CLASS_PATTERN = "^(Test.*|.+[.$]Test.*|.*Tests?)$";
    private static final String IT_CLASS_PATTERN = "^(IT.*|.+[.$]IT.*|.*ITs?)$";

    public static class Suite {
        // TODO JUnit Suite repeats would be sweet! Then we wouldn't need com.github.justincranford.JUnit.Cli.*.

        @org.junit.platform.suite.api.Suite
        @SelectPackages(DEFAULT_PACKAGE)
        @IncludeClassNamePatterns({TEST_CLASS_PATTERN, IT_CLASS_PATTERN})
        public static class All { }

        @org.junit.platform.suite.api.Suite
        @SelectPackages(DEFAULT_PACKAGE)
        @IncludeClassNamePatterns(TEST_CLASS_PATTERN)
        public static class Test { }

        @org.junit.platform.suite.api.Suite
        @SelectPackages(DEFAULT_PACKAGE)
        @IncludeClassNamePatterns(IT_CLASS_PATTERN)
        public static class IT { }
    }

    public static class Cli {
        private static final int DEFAULT_WARMUPS = 1;
        private static final int DEFAULT_REPEATS = 2;
        public static class All {
            public static void main(final String[] args) {
                Utils.cli(Utils.parseIntArg(args, 0, DEFAULT_WARMUPS), Utils.parseIntArg(args, 1, DEFAULT_REPEATS), DiscoverySelectors.selectPackage(DEFAULT_PACKAGE));
            }
        }
        public static class Package {
            public static void main(final String[] args) {
                Utils.cli(Utils.parseIntArg(args, 1, DEFAULT_WARMUPS), Utils.parseIntArg(args, 2, DEFAULT_REPEATS), DiscoverySelectors.selectPackage(args[0]));
            }
        }
        public static class Class {
            public static void main(final String[] args) throws Exception {
                Utils.cli(Utils.parseIntArg(args, 1, DEFAULT_WARMUPS), Utils.parseIntArg(args, 2, DEFAULT_REPEATS), DiscoverySelectors.selectClass(java.lang.Class.forName(args[0])));
            }
        }
        public static class Method {
            public static void main(final String[] args) {
                Utils.cli(Utils.parseIntArg(args, 1, DEFAULT_WARMUPS), Utils.parseIntArg(args, 2, DEFAULT_REPEATS), DiscoverySelectors.selectMethod(args[0]));
            }
        }

        ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

        private static class Utils {
            private static int parseIntArg(final String[] args, final int index, final int defaultValue) {
                return isNullOrEmpty(args) || (index >= args.length) || isEmpty(args[index]) ? defaultValue : Integer.parseInt(args[index]);
            }
            public static void cli(final int warmups, final int repeats, final DiscoverySelector... discoverySelectors) {
                // @see https://junit.org/junit5/docs/current/api/org.junit.platform.launcher/org/junit/platform/launcher/core/LauncherDiscoveryRequestBuilder.html
                final Launcher launcher = LauncherFactory.create();
                final LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
                        .selectors(Arrays.asList(discoverySelectors))
                        .filters(ClassNameFilter.includeClassNamePatterns(TEST_CLASS_PATTERN, IT_CLASS_PATTERN))
                        .build();
                final TestPlan testPlan = LauncherFactory.create().discover(request);
                System.out.println("TESTS FOUND");
                for (final TestIdentifier rootTestIdentifier : testPlan.getRoots()) {
                    recursivePrint("  ", testPlan, rootTestIdentifier);
                }
                System.out.println(" ");
                final String warmupsSummary = loop("WARMUP", warmups, launcher, request);
                System.out.println(warmupsSummary);

                final String repeatsSummary = loop("REPEAT", repeats, launcher, request);
                System.out.println(warmupsSummary + repeatsSummary);
            }

            private static String loop(final String type, final int repeats, final Launcher launcher, final LauncherDiscoveryRequest request) {
                final StringBuilder sb = new StringBuilder("\nSUMMARY OF ").append(type).append("S:\n");
                final long totalNanos = System.nanoTime();
                for (final int repeat : IntStream.rangeClosed(1, repeats).toArray()) {
                    final long repeatNanos = System.nanoTime();
                    try {
                        launcher.execute(request);
                    } finally {
                        final String msg = type + ": " + repeat + " of " + repeats + " [" + ((System.nanoTime() - repeatNanos) / 1000000F) + " msec]";
                        sb.append("  ").append(msg).append('\n');
                        System.out.println(msg);
                    }
                }
                final float totalMillis = (System.nanoTime() - totalNanos) / 1000000F;
                final float averageMillis = totalMillis / repeats;
                final String msg = type + "S: " + repeats + " total=[" + totalMillis + " msec], average=[" + averageMillis + " msec]\n";
                return sb.append(msg).toString();
            }

            private static void recursivePrint(final String indent, final TestPlan testPlan, final TestIdentifier testIdentifier) {
                for (final TestIdentifier childTestIdentifier : testPlan.getChildren(testIdentifier)) {
                    System.out.println(indent + childTestIdentifier.getType() + ": " + childTestIdentifier.getSource() + ", tags=" + childTestIdentifier.getTags());
                    recursivePrint(indent + "   ", testPlan, childTestIdentifier);
                }
            }
        }
    }
}

[jcranfordupgrade]

@sbrannen thanks for the edit. Do you know if repeat annotation for @suite will be reconsidered?