Throwing SkipException should not fail the test with TestNG

[klubi]

Consider below example:
Example uses following dependencies:
spring-test:5.3.3
spring-context:5.3.3
spring-core:5.3.3
testng:7.3.0

package com;

import com.SkipTest.SkipTestConfig;
import com.SkipTest.SkipTestListener;
import org.springframework.context.annotation.Configuration;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.testng.AbstractTestNGSpringContextTests;
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
import org.testng.SkipException;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@ContextConfiguration(classes = SkipTestConfig.class)
@Listeners(SkipTestListener.class)
public class SkipTest extends AbstractTestNGSpringContextTests {

    @Test
    public void shouldNotBeExecuted() {
        System.out.println("EXECUTED");
    }

    @Configuration
    public static class SkipTestConfig {

    }

    public static class SkipTestListener implements IInvokedMethodListener {
        @Override
        public void beforeInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
            throw new SkipException("skip");
        }

        @Override
        public void afterInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
            System.out.println(iTestResult.getStatus());
        }
    }
}

When extends AbstractTestNGSpringContextTests is added then test fails because throwing SkipException in listener method is treated as any other exception.
When extends AbstractTestNGSpringContextTests is removed, then shouldNotBeExecuted is skipped as expected.

Use case:
My system level tests use TestNG listener mechanism to perform several actions, one of which is deciding if given system test can be executed. If test should not be executed then we mark it as skipped and throw SkipException.
This worked flawlessly for TestNG versions 6.+ for version 7.+ it does not, and the issue seems to be in AbstractTestNGSpringContextTests.

[sbrannen]

Thanks for raising the issue.

It took me a while to track it down, but it turns out that this has nothing to do with Spring.

For example, the following test class passes with TestNG 6.11 (with the test method properly skipped) and starts fails with TestNG 6.13.1 or higher.

package com.example.testng_skip;

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
import org.testng.SkipException;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

import com.example.testng_skip.TestNGSkippedTests.MySkipTestListener;

@Listeners(MySkipTestListener.class)
public class TestNGSkippedTests {

	@BeforeMethod
	void beforeMethod() {
		System.err.println(">>>> before ");
	}

	@Test
	void shouldNotBeExecuted() {
		System.err.println(">>>> test ");
	}

	public static class MySkipTestListener implements IInvokedMethodListener {

		@Override
		public void beforeInvocation(IInvokedMethod invokedMethod, ITestResult testResult) {
			throw new SkipException("skip");
		}

		@Override
		public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
		}
	}
}

If you comment out the @BeforeMethod method, the test method will be skipped as expected.

So, there is a regression in TestNG itself.

[sbrannen]

So, there is a regression in TestNG itself.

In light of that, I am closing this issue.

I believe this is effectively a duplicate of testng-team/testng#1632.

[sbrannen]

@klubi, for future reference, whenever you are aware of a related issue open against TestNG (i.e., testng-team/testng#1632), it would be very helpful to us if you could mention that issue in advance so that we may better inform ourselves of possible causes before spending a lot of time investigating on our own.

Thanks

[klubi]

Yeah... @sbrannen I should have :( sorry, will do better next time.