JUnit 5 ParameterizedTest creates single approval file for all test cases

[beatngu13]

Given these dependencies:

<dependency>
	<groupId>org.junit.jupiter</groupId>
	<artifactId>junit-jupiter</artifactId>
	<version>5.7.0</version>
	<scope>test</scope>
</dependency>

<dependency>
	<groupId>com.approvaltests</groupId>
	<artifactId>approvaltests</artifactId>
	<version>9.3.0</version>
	<scope>test</scope>
</dependency>

And this test:

class MyParameterizedApprovalTest {

	enum MyEnum {
		FOO, BAR
	}

	@ParameterizedTest
	@EnumSource(MyEnum.class)
	void testWithEnumSource(MyEnum e) {
		Approvals.verify(e);
	}

	@ParameterizedTest
	@ValueSource(strings = {"foo", "bar"})
	void testWithValueSource(String s) {
		Approvals.verify(s);
	}

}

Creates a single approval file for all test cases of a parameterized test:

The content is the result of the last parameter, i.e. "bar" for testWithValueSource and "BAR" for testWithEnumSource.

My assumption was that this has been fixed as part of #36 / #90 (see JUnit5StackTraceNamerTest)?

[LarsEckart]

hello Daniel,

I remember the tests we wrote in JUnit5StackTraceNamerTest and looking at them, I'd also expect 4 files,
MyParameterizedApprovalTest.testWithEnumSource.FOO.txt
MyParameterizedApprovalTest.testWithEnumSource.BAR.txt
MyParameterizedApprovalTest.testWithValueSource.foo.txt
MyParameterizedApprovalTest.testWithValueSource.bar.txt
but it's just behaving as it normally does for non parameterized tests, as you showed.

While it's not a fix to your solution, what came first to my mind when looking at the combination of parameterized tests with approvalTests, are you aware of verifyAll?

import java.util.List;
import org.approvaltests.Approvals;
import org.junit.jupiter.api.Test;

class MyParameterizedApprovalTest {

  enum MyEnum {
    FOO, BAR
  }

  @Test
  void testWithEnumSource() {
    Approvals.verifyAll("enum source", MyEnum.values());
  }

  @Test
  void testWithValueSource() {
    Approvals.verifyAll("list of strings", List.of("foo", "bar"));
  }
}

I think that would be the approval tests way of testing multiple inputs.

[beatngu13]

Hey @LarsEckart,

Thx for the quick response!

The actual tests are a bit more complicated and each parameter is used as an input value for the class under test. While there are ways to deal with this, a problem is that using @Test results in a single test case, whereas @ParameterizedTest creates a separate test case for every parameter. This influences, e.g., how test results are being displayed within the IDE. When verifying all parameters withing a single test case, it is usually harder to pinpoint which parameter caused a test failure.

For this and some other reasons, I guess verify is better suited than verifyAll for parameterized tests?

[LarsEckart]

Yes, you lose some of the visibility on IDE level for example.
If you print your results from verifyAll nicely, for example each entry in their own line, it's also easy to spot what individual cases changed and thus made the test failed, but you won't see that from the IDE, rather the diff viewer then on test failure.
On Monday we'll have a look at this issue.

[LarsEckart]

hello @beatngu13,

please have a look here: https://github.com/approvals/ApprovalTests.Java/blob/master/approvaltests/docs/how_to/ParameterizedTest.md#top

released in approval tests 9.5.0

the withParameters is almost an alias to something that was already present before but we gave it a nicer name and also support for multiple parameters.

Does that work for you?

[beatngu13]

beatngu13/pdf-zoom-wizard@f78ef9e

Works great—thank you!

Bonus question: Is there a way to "normalize" the parameters before they are being used for the actual filename? For instance, instead of:

TestClass.testMethod.Some param.approved.txt

One might wants to use:

TestClass.testMethod.some_param.approved.txt

It would be nice to have some customization capabilities like in e.g. JUnit Jupiter's display name generator API.