Retries don't work correctly with DataProvider tests

[Tygerian]

I have a IRetryAnalyzer implementation to retry a test up to 3 times for each parameter value. This worked fine up to and including TestNG 6.8.14, but no longer works after that.

In versions 6.8.15-6.8.21 the retry analyzer flat out did not work (see #606).

In 6.9.4 I get way too many retries with the same code.

In 6.9.9 it's a bit better but the tests are still run too many times, even if they pass. Looks like there is some additional retry mechanism that gets triggered after all expected retry attempts are done for each parameter.

See attached console output. I expect to get results seen in 6.8.14.txt. If the mechanism I was using is not longer supported, could you please advise on what is now the correct approach?

Sample code:

public class TestTest {
    @DataProvider
    public Object[][] provider() {
        return new Object[][] {
            {"a"},
            {"b"},
            {"c"},
            {"d"}
        };
    }

    @Test(dataProvider = "provider", retryAnalyzer = MyRetryAnalyzer.class)
    public void test1(String param) {
        System.out.println("Running test1 with param '" + param + "'");
        Assert.assertEquals(param, "c");
    }
}

public class MyRetryAnalyzer implements IRetryAnalyzer {
    private static final int MAX_RETRY_COUNT = 3;

    private final Map<Integer, AtomicInteger> counts = new HashMap<>();

    private AtomicInteger getCount(ITestResult result) {
        int id = Arrays.hashCode(result.getParameters());
        AtomicInteger count = counts.get(id);
        if (count == null) {
            count = new AtomicInteger(MAX_RETRY_COUNT);
            counts.put(id, count);
        }
        return count;
    }

    @Override
    public boolean retry(ITestResult result) {
        int retriesRemaining = getCount(result).getAndDecrement();
        System.out.println(String.format(
                "Retries remaining for test '%s' with params '%s': %d",
                result.getMethod().getMethodName(),
                Arrays.toString(result.getParameters()), retriesRemaining));
        return retriesRemaining > 0;
    }
}

[Tygerian]

6.8.14.txt
6.9.4.txt
6.9.9.txt

[Tygerian]

Might be related to and/or get fixed by #740.

[juherr]

Could you try the branch from #740?
If it fixes you problem, it will be a good motivation to close the two PR together :)

[Tygerian]

Tried with #740 and I guess it does what it's supposed to - the successful test is no longer re-run. However the number of times failed tests are re-tried is incorrect. Also it seems that the retry analyzer is sometimes triggered more than once per failure.

Please see attached output for details.
PR740.txt

[ryanlevell]

Any update on when this could be merged? I am using 6.9.10 and each successive iteration of a data provider gets executed one more time than the last.

For example:

Data Provider	Iterations	Retry Count	Test Runs	Test Fails	Test Skips	Explanation
{ {"a"} }	1	1	2	1	1	Seems correct
{ {"a"}, {"b"} }	2	1	5	3	2	test("b") runs 1 extra time and fails, expected 4 runs not 5
{ {"a"}, {"b"}, {"c"} }	3	1	9	6	3	test("c") runs 2 extra times and fails, expected 6 runs not 9

This pattern continues for each additional iteration added to the data provider. As mentioned in #740, 6.8.14 works correctly.
Sample code below:

public class TmpTest implements IRetryAnalyzer{

  @Test(dataProvider = "dp", retryAnalyzer=TmpTest.class)
  public static void test(String i) {
    Assert.fail();
  }

  @DataProvider(name = "dp")
  public Object[][] dp() {
    // should be 6 total runs: 3 initial attempts and 3 retries
    return new Object[][] { {"a"}, {"b"}, {"c"} };
  }

  // simple way to keep track of retried tests
  List<String> retried = new ArrayList<>();

  @Override
  public boolean retry(ITestResult arg0) {
    if(retried.contains((String)arg0.getParameters()[0])) {
      return false;
    } else {
      retried.add((String)arg0.getParameters()[0]);
      return true;
    }
  }
}

[juherr]

@ryanlevell Could you check if #919 is fixing the issue? Then could you try to complete the pull-request with your provided example?

[aksharpanchal]

This issue is still observed in testng 6.9.13.8 and I also see open issues regarding this #1166

[jmresler]

Is the retry functionality intended for more than one attempt? I've built a suite of tests to retry more than once if failed. I'm seeing issues with one test that falls in the middle of the boundary conditions [0, 3] test with 2 attempts and a randomized failure mechanism does not stop the testng/surefire combination.

I'm always returning either ITestResult.FAILURE or ITestResult.SUCCESS but it still hangs.

[juherr]

@jmresler What do you mean? retryAnalyzer + invocationCount at the same time? I think it is not possible the mix both of them for the moment.