Set/preserve class execution order when using @Factory to create tests

[tatery]

TestNG Version

TestNg v 6.11

Expected behavior

I'd like to have a possibility to preserve/set class execution order to the same order as it is in data from @DataProvider used in @Factory.

Actual behavior

I'd like to execute my test using @Factory that gets data from @DataProvider and I noticed that test instances created by TestNG are executed in random order.
Test class constructor is invoked in the same order as data from data provider so I would expect that the same order should be in class execution but it seems that classes are invoked in random order.

I'm not sure if this is a bug or intended behavior or some configuration is missing in my code. If this is intended behavior then could you please provide some mechanism to set tests instances execution order to the same as data from DataProvider?

Below test code is a simplified version of my original test setup. Originally I provide test data to @DataProvider createTestCmds() from XML file and testng.xml has the following configuration:

<suite name="Basic tests" thread-count="1" group-by-instances="true">
<test name="basic test" group-by-instances="true">
<classes>
<class name="test.TestFactory" />
</classes>
</suite>

Is the issue reproductible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

public class TestFactory {

	private String cmd = null;

	@DataProvider
	public static Iterator<Object[]> createTestCmds() {
		ArrayList<Object[]> testData = new ArrayList<Object[]>(9);
		for(int i = 0; i < 6; i++) {
			String test = "TEST " + i;
			System.out.println("Command in data provider: " + test);
			Object[] ob = new Object[] { test };
			testData.add(ob);
		}
		return testData.iterator();
	}

	@Factory(dataProvider = "createTestCmds")
	public TestFactory(String cmd) {
		this.cmd = cmd;
		System.out.println("Command in constructor: " + this.cmd);
	}

	@DataProvider
	// this is not needed however data from providers are nicely added to test log
	public Object[][] cmdProvider() {
		return new Object[][] { { this.cmd } };
	}

	@Test(dataProvider = "cmdProvider")
	public void sendCmd(String cmd) {
		System.out.println("Command to send: " + cmd);
	}
}

Test output:

Commands order in data provider

 TEST 0
 TEST 1
 TEST 2
 TEST 3
 TEST 4
 TEST 5

Commands order in test factory method (test class constructor):

 TEST 0
 TEST 1
 TEST 2
 TEST 3
 TEST 4
 TEST 5

Commands execution order (each execution produce different order):

 TEST 3
 TEST 2
 TEST 5
 TEST 1
 TEST 4
 TEST 0

[juherr]

The behavior is strange and should be the same with or without @Test(dataProvider = "cmdProvider").

Thanks for the report.

[tatery]

Hi,

This issue is really annoying so I wonder if you know any workaround that I could apply before you fix (or not) this issue. Thank you in advance for any advice.

@juherr thanks for description formatting.

[hemano]

@juherr
Do we have any workaround for now to maintain the order.
I understand that below 2 issues where talking about the same.

Note: I've tried using version 6.12 of testng. The behaviour is same.

#1454
#1506

[hemano]

@tatery
You can use interface IMethodInterceptor to solve your issue.

http://www.seleniumeasy.com/testng-tutorials/imethodinterceptor-example-to-reorder-tests

[krmahadevan]

@hemano - An IMethodInterceptor will not do the trick here, because its invoked only once per TestNG execution and it is handed over all the test methods in one shot. I guess what is being asked here by @tatery is a way in which one can order the test methods within the test class instances, in the same order in which the test class was instantiated.

@tatery - Can you please try doing the following

• Upgrade to TestNG v6.12 (the latest version of TestNG)
• Add a toString() implementation to your TestFactory class, which would be based on the parameters that are being injected to your test class's constructor

Now TestNG would start ordering your instances based on the toString() implementation.

Here's a full fledged elaborate example that demonstrates what I am talking about

import org.testng.TestNG;
import org.testng.annotations.AfterTest;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.Comparator;
import java.util.Iterator;
import java.util.List;

public class TestFactory {
    private String cmd = null;
    private static final List<String> messages = new ArrayList<>();

    public static void main(String[] args) {
        TestNG testng = new TestNG();
        XmlSuite xmlSuite = new XmlSuite();
        xmlSuite.setGroupByInstances(true);
        xmlSuite.setName("Sample_Test_Suite");
        XmlTest xmlTest = new XmlTest(xmlSuite);
        xmlTest.setName("Sample_Test");
        xmlTest.setClasses(Collections.singletonList(new XmlClass(TestFactory.class)));
        testng.setXmlSuites(Collections.singletonList(xmlSuite));
        testng.setVerbose(2);
        System.err.println("Printing the suite xml file that would be used.");
        System.err.println(xmlSuite.toXml());
        testng.run();
    }

    @DataProvider
    public static Iterator<Object[]> createTestCmds() {
        final List<Object[]> testData = new ArrayList<>(9);
        //Purposefully adding elements in the reverse order.
        for (int i = 6; i >= 0; i--) {
            String test = "TEST " + i;
            Object[] ob = new Object[]{test};
            testData.add(ob);
        }
        //Now sorting the elements in the order just for demonstration purposes.
        Collections.sort(testData, new Comparator<Object[]>() {
            @Override
            public int compare(Object[] o1, Object[] o2) {
                if (o1 == null || o2 == null || o1.length != o2.length) {
                    return 0;
                }
                return o1[0].toString().compareTo(o2[0].toString());
            }
        });
        Iterator<Object[]> iterator = new LoggingIterator(testData.iterator());
        return iterator;
    }

    @Factory(dataProvider = "createTestCmds")
    public TestFactory(String cmd) {
        this.cmd = cmd;
        messages.add("Command in constructor: " + this.cmd);
    }

    @DataProvider
    // this is not needed however data from providers are nicely added to test log
    public Object[][] cmdProvider() {
        return new Object[][]{{this.cmd}};
    }

    @Test(dataProvider = "cmdProvider")
    public void sendCmd(String cmd) {
        messages.add("Command to send: " + cmd);
    }

    @Test
    public void anotherCommand() {
        messages.add("Another Command: " + cmd);
    }

    @AfterTest
    public void printAllMessages() {
        for (String message : messages) {
            System.err.println(message);
        }
    }

    @Override
    public String toString() {
        return this.cmd;
    }

    public static class LoggingIterator implements Iterator<Object[]> {
        private Iterator<Object[]> iterator;

        public LoggingIterator(Iterator<Object[]> iterator) {
            this.iterator = iterator;
        }

        @Override
        public boolean hasNext() {
            return iterator.hasNext();
        }

        @Override
        public Object[] next() {
            Object[] data = iterator.next();
            messages.add("Producing cmd: " + Arrays.toString(data));
            return data;
        }
    }
}

Now when the main() method is executed, the below output would be seen

Printing the suite xml file that would be used.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Sample_Test_Suite" group-by-instances="true">
  <test name="Sample_Test">
    <classes>
      <class name="com.rationaleemotions.github.issue1410.TestFactory"/>
    </classes>
  </test> <!-- Sample_Test -->
</suite> <!-- Sample_Test_Suite -->

Producing cmd: [TEST 0]
Command in constructor: TEST 0
Producing cmd: [TEST 1]
Command in constructor: TEST 1
Producing cmd: [TEST 2]
Command in constructor: TEST 2
Producing cmd: [TEST 3]
Command in constructor: TEST 3
Producing cmd: [TEST 4]
Command in constructor: TEST 4
Producing cmd: [TEST 5]
Command in constructor: TEST 5
Producing cmd: [TEST 6]
Command in constructor: TEST 6
Another Command: TEST 0
Command to send: TEST 0
Another Command: TEST 1
Command to send: TEST 1
Another Command: TEST 2
Command to send: TEST 2
Another Command: TEST 3
Command to send: TEST 3
Another Command: TEST 4
Command to send: TEST 4
Another Command: TEST 5
Command to send: TEST 5
Another Command: TEST 6
Command to send: TEST 6
PASSED: anotherCommand on TEST 0
PASSED: sendCmd on TEST 0("TEST 0")
PASSED: anotherCommand on TEST 1
PASSED: sendCmd on TEST 1("TEST 1")
PASSED: anotherCommand on TEST 2
PASSED: sendCmd on TEST 2("TEST 2")
PASSED: anotherCommand on TEST 3
PASSED: sendCmd on TEST 3("TEST 3")
PASSED: anotherCommand on TEST 4
PASSED: sendCmd on TEST 4("TEST 4")
PASSED: anotherCommand on TEST 5
PASSED: sendCmd on TEST 5("TEST 5")
PASSED: anotherCommand on TEST 6
PASSED: sendCmd on TEST 6("TEST 6")

===============================================
    Sample_Test
    Tests run: 14, Failures: 0, Skips: 0
===============================================


===============================================
Sample_Test_Suite
Total tests run: 14, Failures: 0, Skips: 0
===============================================


Process finished with exit code 0

As you can see from the output, the order of execution in general is as below

1. Data provider is invoked for one set of data (Since we have involved Iterator, we are resorting to lazy data providers approach)
2. The produced data from (1) is used to create 1 instance of the test class TestFactory
3. (1) and (2) is repeated 7 times (because that is how much data is returned by the data provider)
4. Now TestNG basically sorts the instances based on the toString() implementation which we provided (This can be seen by commenting out the Collections.sort(), wherein our data provider produces values of TEST 6, TEST 5, TEST 4 and so on.. but when it comes to TestNG invoking the @Test methods on the instances, it starts off with TEST 0, TEST 1 and so on.