Performance issue: @DataPoints from functions and @Theory with multiple arguments

[dev930018]

This is still an issue as of 4.12 and can cause a huge unexpected performance hit.

Minimal example

For N=5 datapoints of type T=int and M=3 arguments of type T to a Theory

import org.junit.runner.RunWith;
import org.junit.experimental.theories.Theory;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.DataPoints;
import static org.junit.Assert.assertTrue;

@RunWith(Theories.class)
public class DataPointsBug {

    @DataPoints
    public static int[] generateDataPoints() {
        System.out.println("generateDataPoints()");
        // Expensive operation
        try {
            Thread.sleep(1000);
        } catch (InterruptedException ex) {
        }
        return new int[] {1,2,3,4,5};
    }

    @Theory
    public void manyArguments(int a, int b, int c) {
        // Cheap test
        assertTrue(a > 0 && b > 0 && c > 0);
    }
}

Output

generateDataPoints()
<line repeated 30 more times>

Finished in 31s 45ms

In general, for N datapoints of type T and a Theory taking M arguments of type T, this requires S = sum(1 + N + N^2 + ... + N^M) ~ O(N^M), which seems pretty excessive...

Workaround

It is an easy-enough workaround to store the generator output once and annotate the variable with @DataPoints:

import org.junit.runner.RunWith;
import org.junit.experimental.theories.Theory;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.DataPoints;
import static org.junit.Assert.assertTrue;

@RunWith(Theories.class)
public class DataPointsBug {

    private static int[] generateDataPoints() {
        System.out.println("generateDataPoints()");
        // Expensive operation
        try {
            Thread.sleep(1000);
        } catch (InterruptedException ex) {
        }
        return new int[] {1,2,3,4,5};
    }
    @DataPoints
    public static int[] dataPoints = generateDataPoints();

    @Theory
    public void manyArguments(int a, int b, int c) {
        // Cheap test
        assertTrue(a > 0 && b > 0 && c > 0);
    }
}

Output

generateDataPoints()

Finished in 1s 37ms

Should this be re-opened and addressed?

Originally posted by @dev930018 in #82 (comment)

[junit-builds]

JUnit 4 is now in maintenance mode.

At this point, only critical bugs and security issues will be fixed.

The team has therefore decided to close this issue.