Conflict beween config failure skips and listener forced exceptions

[baubakg]

TestNG Version

7.5
(regression in regards to 7.4.0)

Expected behavior

If you throw a skip exception in one of your listeners whilst a BeforeX method has already failed, the state of the tests should be skipped (in 7.4.0 it was the case. The new skip overrode the configure issue)

Also we should not lose parameter/data provider data.

Actual behavior

Since 7.5 (Worked fine in 7.4.0 (despite #2522 )) If you throw a skip exception in one of your listeners whilst a BeforeX method has already failed, the state of the tests will switch to "Failed".

Another consequence of this is that if you use a data provider, the parameter will be lost in the ITestResult object .

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

In my examples bellow, My listener, DP_Listeners, throws a skip exception.

My test class MyTest has a beforeSuite method that contains a failing assertion.

(I have omitted my helper methods, but can add them if you need it)

   @Test
   public void problemTest() {

       // Rampup
       TestNG myTestNG = createTestNG();
       TestListenerAdapter tla = fetchTestResultsHandler(myTestNG);

       // Define suites
       XmlSuite mySuite = addSuitToTestNGTest(myTestNG, "Problem Test Suite");

       // Add listeners
       mySuite.addListener(DP_Listeners.class.getTypeName());

       // Create an instance of XmlTest and assign a name for it.
       XmlTest myTest = attachTestToSuite(mySuite, "Test config methods");

       myTest.setXmlClasses(Arrays.asList(new XmlClass(MyTest.class)));

       myTestNG.run();

       SoftAssert softAssertion = new SoftAssert();
       softAssertion.assertEquals(tla.getPassedTests().size(), 0, "No tests should be passed");
       softAssertion.assertEquals(tla.getSkippedTests().size(), 2, "All tests tests should be skipped");
       softAssertion.assertEquals(tla.getFailedTests().size(), 0, "No tests tests should be failed");
       
       if (tla.getFailedTests().size()>0) {
           softAssertion.assertEquals(tla.getFailedTests().get(0).getParameters().length, 1, "We should still keep our parameters");
       }
       softAssertion.assertAll();
   }    

My Listener

public class DP_Listeners implements ITestListener {

    @Override
    public void onTestStart(ITestResult result) {
        if (result.getName().equals("simpleTest1")) {
            throw new SkipException("Forced Skip");
        }
    }

}

My target test class:

public class MyTest {
    @BeforeSuite
    public void beforePhasedSuite() {
        int beforeValue = 13;

        Assert.assertEquals(beforeValue, 14);
    }
    
    @DataProvider (name = "myProvider")
    public Object[][] dpMethod() {
        return new Object [][] { {"P1"}};
    }
    
    @Test(dataProvider = "myProvider")
    public void simpleTest1(String s) {
        System.out.println("In test1 "+s);
    }
    
    @Test(dataProvider = "myProvider")
    public void simpleTest2(String s) {
        System.out.println("In test2 "+ s);
    }

}

[krmahadevan]

@baubakg - I was not able to reproduce the error using 7.4.0. Please find attached a sample project for the same. Can you please confirm ?
github_2721.zip