New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
successPercentage attribute of @Test does not work correctly #309
Comments
This is a very old issue but still standing, just tested locally. Any idea if is going to be addressed or any workaround? |
@cbeust : this is biting us when we use TestNG together with Jenkinsfile-based builds. Any thoughts about how hard it'd be to fix, or appetite for accepting a PR? |
Always happy to accept a PR, Patrick! |
I am also seeing behavior where tests that succeed below the specified success % are not causing a failure in the suite. Currently using TestNG v6.10 This seems like a fairly old ticket, maybe there has been a fix for this since it's been open? Are there any plans to fix this? Thanks |
Revisited this issue using TestNG Java class package com.rationaleemotions;
import org.testng.Assert;
import org.testng.annotations.Test;
public class RepeatTest {
int count = 0;
@Test
public void normal() {
System.out.println("repeat..." + (++count));
}
@Test(invocationCount = 30, successPercentage = 98)
public void successRate() {
System.out.println("repeat..." + (++count));
if (count > 15) {
Assert.fail("fail with count > 15");
}
}
} suite file <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="309_suite" parallel="false" configfailurepolicy="continue">
<test name="309_test">
<classes>
<class name="com.rationaleemotions.RepeatTest"/>
</classes>
</test>
</suite> Output
Test project that was used. |
I'm using testng-6.8, while running a multiple invocation test, I found that the "successPercentage" of @test annotation is not working correctly.
Reproducer:
===build.xml===
===repeat.config.xml===
===RepeatTest.java===
According to http://testng.org/doc/documentation-main.html#annotations, if "successPercentage" is specified, testng should fail the test if pass percentage is smaller than expected. However, if you run above reproducer, you will see it prints below output, but never sends a FAILURE event to listeners. It only works when percentage is set to 100, or is not set.
This seems like a regression, I did no see the issue in testng-5.8.
===output===
Notice that failure count in repeat-test section is 0, and it did not log any FAILURE even it does not meet the expected percentage.
Regards,
Gan
The text was updated successfully, but these errors were encountered: