New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(InvocationCount or DataProvider) and RetryAnaylzer when used togther performs incorrect behaviour #871
Comments
shroffsagar
changed the title
InvocationCount and RetryAnaylzer when used togther performs incorrect behaviour
(InvocationCount or DataProvider) and RetryAnaylzer when used togther performs incorrect behaviour
Nov 29, 2015
I have tried reproducing this in TestNG Using invocation counts with retry analyzerimport java.util.concurrent.atomic.AtomicInteger;
import org.testng.Assert;
import org.testng.IExecutionListener;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
@Listeners(CountingListener.class)
public class InvocationCountPoweredSampleTestCase {
private static final AtomicInteger counter = new AtomicInteger(0);
@Test(retryAnalyzer = OnceMore.class, invocationCount = 2)
public void testMethod() {
counter.incrementAndGet();
int currentInvocationCount = Reporter.getCurrentTestResult().getMethod()
.getCurrentInvocationCount();
System.err.println(getClass().getName() + ".testMethod(): Counter = " + counter.get() +", Invocation Count = " + currentInvocationCount);
Assert.fail();
}
public static class OnceMore implements IRetryAnalyzer {
private final AtomicInteger count = new AtomicInteger(0);
@Override
public boolean retry(ITestResult result) {
int currentInvocationCount = result.getMethod().getCurrentInvocationCount();
int current = count.incrementAndGet();
System.err.println(getClass().getEnclosingClass().getName() + ".retry(): Counter = " + counter.get() +", Invocation Count = " + currentInvocationCount);
return current < 2;
}
}
public static class CountingListener implements IExecutionListener {
@Override
public void onExecutionFinish() {
System.err.println(getClass().getEnclosingClass().getName() + ": Total count for testMethod() is " + counter.get());
}
}
} Output com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(): Counter = 1, Invocation Count = 0
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.retry(): Counter = 1, Invocation Count = 0
Test ignored.
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(): Counter = 2, Invocation Count = 1
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.retry(): Counter = 2, Invocation Count = 1
java.lang.AssertionError: null
at com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(InvocationCountPoweredSampleTestCase.java:24)
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(): Counter = 3, Invocation Count = 2
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.retry(): Counter = 3, Invocation Count = 2
java.lang.AssertionError: null
at com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(InvocationCountPoweredSampleTestCase.java:24)
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(): Counter = 4, Invocation Count = 3
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.retry(): Counter = 4, Invocation Count = 3
java.lang.AssertionError: null
at com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase.testMethod(InvocationCountPoweredSampleTestCase.java:24)
===============================================
Default Suite
Total tests run: 4, Passes: 0, Failures: 3, Skips: 0, Retries: 1
===============================================
com.rationaleemotions.github.issue871.InvocationCountPoweredSampleTestCase: Total count for testMethod() is 4
Process finished with exit code 0
Notice this line in the above output:
For every test invocation's failure, the retry happens twice. So total invocations are 2*2 Using data provider with retry analyzerimport java.util.concurrent.atomic.AtomicInteger;
import org.testng.Assert;
import org.testng.IExecutionListener;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
@Listeners(CountingListener.class)
public class DataProviderPoweredSampleTestCase {
private static final AtomicInteger counter = new AtomicInteger(0);
@Test(retryAnalyzer = OnceMore.class, dataProvider = "dp")
public void testMethod(int ignored) {
counter.incrementAndGet();
int currentInvocationCount = Reporter.getCurrentTestResult().getMethod()
.getCurrentInvocationCount();
System.err.println(
getClass().getName() + ".testMethod(): Counter = " + counter.get() + ", Invocation Count = "
+ currentInvocationCount);
Assert.fail();
}
@DataProvider(name = "dp")
public Object[][] getTestData() {
return new Object[][]{
{1},
{2}
};
}
public static class OnceMore implements IRetryAnalyzer {
private final AtomicInteger count = new AtomicInteger(0);
@Override
public boolean retry(ITestResult result) {
int currentInvocationCount = result.getMethod().getCurrentInvocationCount();
int current = count.incrementAndGet();
System.err.println(
getClass().getEnclosingClass().getName() + ".retry(): Counter = " + counter.get()
+ ", Invocation Count = " + currentInvocationCount);
return current < 2;
}
}
public static class CountingListener implements IExecutionListener {
@Override
public void onExecutionFinish() {
System.err.println(
getClass().getEnclosingClass().getName() + ": Total count for testMethod() is "
+ counter.get());
}
}
} Output com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(): Counter = 1, Invocation Count = 0
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.retry(): Counter = 1, Invocation Count = 0
Test ignored.
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(): Counter = 2, Invocation Count = 1
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.retry(): Counter = 2, Invocation Count = 1
java.lang.AssertionError: null
at com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(DataProviderPoweredSampleTestCase.java:27)
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(): Counter = 3, Invocation Count = 2
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.retry(): Counter = 3, Invocation Count = 2
Test ignored.
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(): Counter = 4, Invocation Count = 3
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.retry(): Counter = 4, Invocation Count = 3
java.lang.AssertionError: null
at com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase.testMethod(DataProviderPoweredSampleTestCase.java:27)
===============================================
Default Suite
Total tests run: 4, Passes: 0, Failures: 2, Skips: 0, Retries: 2
===============================================
com.rationaleemotions.github.issue871.DataProviderPoweredSampleTestCase: Total count for testMethod() is 4
Process finished with exit code 0
Please note the below line in the above output
If this is still a problem, please help share a reproducible sample and comment. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Below are the steps to reproduce both the issue.
For data provider issue, one needs to use a data provider with testinputs same as invocationCount.
Above configures my RetryAnalyzer to max retry attempts to 3. And total invocationCount to 2
Below is the output
As noticed in the above output. Test is invoked more then twice.
The total invocation goes wrong only when a retry is invoked.
The current invocation depends on the total times the method is invoked. The count is cumulative with each new iteration.
Please review below examples. All the retry fails in below examples
The text was updated successfully, but these errors were encountered: