Document scope of applicability for TestWatcher implementations

[wazzeps]

Referring to #2560

Steps to reproduce

	@Disabled
	@ParameterizedTest
	@ValueSource(strings = {"a", "b"})
	void testParam(String s) {
		System.out.println(s);
	}

With TestWatcher implementation

	@Override
	public void testDisabled(ExtensionContext context, Optional<String> reason) {
		System.out.println("DISABLED");
	}

Expected: 2 outputs "DISABLED"
Actual: no output, testDisabled method wasn't called

Context

• junit-jupiter-api:5.9.2
• junit-jupiter-engine:5.9.2
• Gradle
• IntelliJ IDEA

[sbrannen]

I have confirmed that disabled events are not supplied to a TestWatcher for @TestTemplate methods (e.g., @RepeatedTest and @ParameterizedTest, and I have changed the title of this issue to reflect that.

The following can be used to reproduce the issue.

class WatcherTests {

	@RegisterExtension
	Extension watcher = new TestWatcher() {

		@Override
		public void testSuccessful(ExtensionContext context) {
			System.err.println("SUCCESSFUL: " + context.getRequiredTestMethod().getName());
		}

		@Override
		public void testDisabled(ExtensionContext context, Optional<String> reason) {
			System.err.println("DISABLED  : " + context.getRequiredTestMethod().getName());
		}
	};

	@Disabled
	@Test
	void test() {
	}

	@Disabled
	@RepeatedTest(2)
	void repeatedTest() {
	}

	@Disabled
	@ParameterizedTest
	@ValueSource(strings = { "a", "b" })
	void parameterizedTest(String s) {
	}

}

Since the documentation states the following, I am labeling this as a bug.

Please note that this API is currently only used to report the results of @test methods and @TestTemplate methods (e.g., @RepeatedTest and @ParameterizedTest).

Thanks for raising the issue, @wazzeps.

[sbrannen]

Oops. I retract that.

Everything works fine if the TestWatcher is registered at the class level.

For example, changing the declaration to static as follows causes the disabled events to be reported as expected.

	@RegisterExtension
	static Extension watcher = new TestWatcher() { // ...

I am therefore repurposing this issue to improve the documentation for TestWatcher to make users of the API aware of the different semantics when a TestWatcher is not registered at the class level.

[wazzeps]

If I understand correctly, using an extension with @ExtendWith in a test class declaration is also registering at the class level.

@ExtendWith(TestWatcherImpl.class)
public class SomeTest {

In this case TestWatcher methods are not called.

[sbrannen]

In this case TestWatcher methods are not called.

I am not able to reproduce that.

When I execute the following, it prints out:

DISABLED  : repeatedTest
DISABLED  : test
DISABLED  : parameterizedTest

@ExtendWith(WatcherTests.MyWatcher.class)
class WatcherTests {

	static class MyWatcher implements TestWatcher{
		@Override
		public void testSuccessful(ExtensionContext context) {
			System.err.println("SUCCESSFUL: " + context.getRequiredTestMethod().getName());
		}

		@Override
		public void testDisabled(ExtensionContext context, Optional<String> reason) {
			System.err.println("DISABLED  : " + context.getRequiredTestMethod().getName());
		}
	}

	@Disabled
	@Test
	void test() {
	}

	@Disabled
	@RepeatedTest(2)
	void repeatedTest() {
	}

	@Disabled
	@ParameterizedTest
	@ValueSource(strings = { "a", "b" })
	void parameterizedTest(String s) {
	}

}

[wazzeps]

In my case, I implemented TestWatcher in a separate class

public class TestWatcherImpl implements TestWatcher {

and then

@ExtendWith(TestWatcherImpl.class)
public class SomeTest {

[sbrannen]

In my case, I implemented TestWatcher in a separate class

That doesn't make a difference. A top-level class exhibits the same behavior as a static nested class.

For example, the following also prints:

DISABLED  : repeatedTest
DISABLED  : test
DISABLED  : parameterizedTest

@ExtendWith(MyWatcher.class)
public class WatcherTests {

	@Disabled
	@Test
	void test() {
	}

	@Disabled
	@RepeatedTest(2)
	void repeatedTest() {
	}

	@Disabled
	@ParameterizedTest
	@ValueSource(strings = { "a", "b" })
	void parameterizedTest(String s) {
	}

}

class MyWatcher implements TestWatcher {
	@Override
	public void testSuccessful(ExtensionContext context) {
		System.err.println("SUCCESSFUL: " + context.getRequiredTestMethod().getName());
	}

	@Override
	public void testDisabled(ExtensionContext context, Optional<String> reason) {
		System.err.println("DISABLED  : " + context.getRequiredTestMethod().getName());
	}
}

[wazzeps]

DISABLED : repeatedTest
DISABLED : test
DISABLED : parameterizedTest

Both of your examples end with DISABLED : parameterizedTest.

Could you please check what happens if the test with @ParameterizedTest runs first?

[sbrannen]

It doesn't make a difference if the parameterized test runs first.

@ExtendWith(WatcherTests.MyWatcher.class)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class WatcherTests {

	static class MyWatcher implements TestWatcher {

		@Override
		public void testSuccessful(ExtensionContext context) {
			System.err.println("SUCCESSFUL: " + context.getRequiredTestMethod().getName());
		}

		@Override
		public void testDisabled(ExtensionContext context, Optional<String> reason) {
			System.err.println("DISABLED  : " + context.getRequiredTestMethod().getName());
		}
	}


	@Order(1)
	@Disabled
	@ParameterizedTest
	@ValueSource(strings = { "a", "b" })
	void parameterizedTest(String s) {
	}

	@Order(2)
	@Disabled
	@Test
	void test() {
	}

	@Order(3)
	@Disabled
	@RepeatedTest(2)
	void repeatedTest() {
	}

}

Here's the result:

DISABLED  : parameterizedTest
DISABLED  : test
DISABLED  : repeatedTest

[wazzeps]

Interesting. I got different results.

@Order(1)
@Disabled
@ParameterizedTest
@ValueSource(strings = { "a", "b" })
void parameterizedTest(String s) {
}

@Order(2)
@Disabled
@Test
void test() {
}

@Order(3)
@Disabled
@RepeatedTest(2)
void repeatedTest() {
}

then

DISABLED  : parameterizedTest
DISABLED  : repeatedTest

@Order(1)
@Disabled
@ParameterizedTest
@ValueSource(strings = { "a", "b" })
void parameterizedTest(String s) {
}

@Order(2)
@Disabled
@RepeatedTest(2)
void repeatedTest() {
}

@Order(3)
@Disabled
@Test
void test() {
}

then

DISABLED : parameterizedTest

[Jusjuc]

If you annotate a test class with @Disabled, the watcher doesn't get invoked for any type of test.

@Disabled
class WatcherTests {

    @RegisterExtension
    static Extension watcher = new TestWatcher() {
        @Override
        public void testSuccessful(ExtensionContext context) {
            System.err.println("SUCCESSFUL: " + context.getRequiredTestMethod().getName());
        }

        @Override
        public void testDisabled(ExtensionContext context, Optional<String> reason) {
            System.err.println("DISABLED  : " + context.getRequiredTestMethod().getName());
        }
    };

    @Test
    void test() {
    }

    @RepeatedTest(2)
    void repeatedTest() {
    }

    @ParameterizedTest
    @ValueSource(strings = { "a", "b" })
    void parameterizedTest(String s) {
    }
}

The output is empty in such case.

[sbrannen]

If you annotate a test class with @Disabled, the watcher doesn't get invoked for any type of test.

Good point. 👍

The documentation already states the following.

if there is a failure at the class level — for example, an exception thrown by a @BeforeAll method — no test results will be reported.

So that's similar in nature to the @Disabled test class scenario, and I'll update the documentation to reflect that.