Parameter-provider method is always executed, even if its test class is excluded

[cwardgar]

JUnit always evaluates a Parameterized test class's @Parameters method, even when its tests will be skipped due to a @Category exclusion. This is problematic for thredds because many of those parameter-provider methods do expensive and unnecessary operations, such as scanning all or part of TestDir.cdmUnitTestDir. They're SUPER spammy in our log output as well.

I'd like to find a test runner that doesn't invoke the parameter-provider unless necessary. Junit's included Parameterized class doesn't fit the bill, but JUnitParams might. Spock might work as well.

[cwardgar]

This is a particular problem on Travis, where cdmUnitTest/ is not available, but those parameter-provider methods are run anyway. The result is a bunch of spurious error messages like:

ucar.nc2.dataset.TestOpenWithEnhanceP STANDARD_OUT
    ---------------Reading directory /home/travis/build/JohnLCaron/thredds/build/NO/unidata.testdata.path/FOUND/cdmUnitTest/conventions
    ---------------INVALID /home/travis/build/JohnLCaron/thredds/build/NO/unidata.testdata.path/FOUND/cdmUnitTest/conventions

ucar.nc2.dataset.TestOpenWithEnhanceP STANDARD_ERROR
    java.io.FileNotFoundException: Cant open /home/travis/build/JohnLCaron/thredds/build/NO/unidata.testdata.path/FOUND/cdmUnitTest/conventions
        at ucar.unidata.test.util.TestDir.actOnAll(TestDir.java:344)
        at ucar.unidata.test.util.TestDir.actOnAllParameterized(TestDir.java:311)
        at ucar.nc2.dataset.TestOpenWithEnhanceP.getTestParameters(TestOpenWithEnhanceP.java:58)

Note that TestOpenWithEnhanceP is marked with @Category(NeedsCdmUnitTest.class), and is thus excluded on Travis. Those messages are still printed.

[cwardgar]

It turns out that JUnitParams has the same behavior as JUnit: the parameter-provider method is always executed, even if its test class is excluded.

A simple solution that just occurred to me is to check the directory pointed to by TestDir.cdmUnitTestDirin the parameter-provider method and return an empty list if it doesn't exist.