Execution order changed from version 6.0.1 to 6.8

[ajay516]

I just upgraded my project (from 6.0.1 to 6.8) and found that the order of execution of tests are changed. In version of 6.0.1, the order of execution is:

Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNGMapConfigurator@72ad8bfd
main     BaseTest(Actual1Test) - beforeClass
main  Actual1Test(Actual1Test) - test
main  Actual1Test(Actual1Test) - test3
main  Actual1Test(Actual1Test) - test4 - one
main  Actual1Test(Actual1Test) - test4 - two
main  Actual1Test(Actual1Test) - test4 - three
main  Actual1Test(Actual1Test) - test4 - four
main     BaseTest(Actual1Test) - afterClass
main     BaseTest(Actual2Test) - beforeClass
main  Actual2Test(Actual2Test) - test
main     BaseTest(Actual2Test) - afterClass
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.894 sec

In version 6.8, the order becomes:

Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@3301b455
main     BaseTest(Actual1Test) - beforeClass
main  Actual1Test(Actual1Test) - test
main  Actual1Test(Actual1Test) - test4 - one
main  Actual1Test(Actual1Test) - test4 - two
main  Actual1Test(Actual1Test) - test4 - three
main  Actual1Test(Actual1Test) - test4 - four
main     BaseTest(Actual2Test) - beforeClass
main  Actual2Test(Actual2Test) - test
main     BaseTest(Actual2Test) - afterClass
main  Actual1Test(Actual1Test) - test3
main     BaseTest(Actual1Test) - afterClass
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.478 sec

Here are my observations:

1. All the tests and @BeforeXXX methods are executed together in older version. In the new version, the execution order is mixed
2. I am creating database in @BeforeClass of base class and tearing it @afterclass methods of BaseClass. When @BeforeClass is called twice, it throws error stating that the database already exists. The codebase I have attached removed that complications and just demonstrates the execution order.
3. I have also observed that at times, the execution order is same as of old one - while I am not able to figure out why.

Here is my execution environment:

Apache Maven 3.0.4 (r1232337; 2012-01-17 14:14:56+0530)
Maven home: /opt/local/share/java/maven3
Java version: 1.7.0_06, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_06.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.7.5", arch: "x86_64", family: "mac"

[ajay516]

Here is the code: https://github.com/ajay516/ajay516-testng-execution-order

[VladRassokhin]

Yeah, i can reproduce that.

Probably workaround: use testng.xml with preserve-order="true".

After some investigations i have found that "dependOnMethods" matter. Without dependency all works fine. Looks like TestNG incorrectly reorders test methods to run.

Simplified code you can find at VladRassokhin/testng-execution-order

[riemenschneider]

Is there a workaround (except removing all dependsOnMethods)?

[ShijunK]

any clue of the root cause? action plan? The reason (at least for my use case:selenium ui testing) to choose testng over junit is tests dependencies.

order all tests in xml is not feasible for large project with hundreds, thousands tests.

[cedricrefresh]

If you want precise ordering, you should use dependsOnGroups, not XML (which was never designed to specify ordering).

[ShijunK]

dependsOnGroups does not work, at least not as my understanding.
with parallel="false", I expect suite runs all tests in single thread, by the order defined either java5 annotations or xml (with preserve-order). depended groups/tests should be executed only once.

given one base class, and two sub classes, depended group in base class

the expected output:

beforeSuite
init
testA1
testA2
testB1
testB2

actual output:

beforeSuite
init
init
testA1
testA2
testB1
testB2

===============================================
Custom suite
Total tests run: 6, Failures: 0, Skips: 0
===============================================

suite.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Custom suite" parallel="false">
    <test name="Test dependencies">
        <classes>
            <class name="SubClassA"/>
            <class name="SubClassB"/>
        </classes>
    </test>
</suite>

public class BaseClass {
    @BeforeSuite
    public void beforeSuite() {
        System.out.println("beforeSuite");
    }

    @Test(groups = {"init"})
    public void init(){
        System.out.println("init");
    }
}
public class SubClassA extends BaseClass {
    @Test(dependsOnGroups = {"init.*"})
    public void testA1() {
        System.out.println("testA1");
    }

    @Test(dependsOnMethods = "testA1")
    public void testA2() {
        System.out.println("testA2");
    }
}
public class SubClassB extends BaseClass{
    @Test(dependsOnGroups = {"init.*"})
    public void testB1() {
        System.out.println("testB1");
    }

    @Test(dependsOnMethods = "testB1")
    public void testB2(){
        System.out.println("testB2");
    }
}

tell me, if I understand the documentation in a wrong way. Appreciate, if anyone can tell me how to properly use test dependencies in testng.

[cbeust]

The ordering looks correct to me. What you seem to be wanting is atomicity (invoking groups of methods together)?

[ShijunK]

how could it be right? per http://testng.org/doc/documentation-main.html#testng-xml
"TestNG will run your tests in the order they are found in the XML file"

my understand is, TestNG executes tests in the order found in the XML file, within each test, annotations (dependsOnGroup, dependsOnMethods) will decide the order.

so, I have SubClassA before SubClassB in xml, I expect TestNG execute SubClassA before doing anything about SubClassB. no?

[cbeust]

The depend annotations override anything that is found in the XML. Either you use preserve-order=true in your XML file or you define dependencies in Java.

[ShijunK]

Thanks for the clarification. One more question, when you say "override anything that is found in the XML", that also apply to "group-by-instances" / "order-by-instances"? There is no way to archive it by using annotations?

btw, should it be "group-by_instances" or "order-by-instances"? Neither of them exists in http://testng.org/testng-1.0.dtd

from http://testng.org/doc/documentation-main.html

For this ordering, you can use the XML attribute group-by-instances. This attribute is valid either on <suite> or <test>:
  <suite name="Factory" order-by-instances="true">
or
  <test name="Factory" order-by-instances="true">

[adimoshu]

Hello,

I am using the latest version of TestNG (6.8.8) and I don't know how to solve an issue that I have.
I have on Jenkins a job that executes some automation tests, These are TestNG tests and they are part of a suite file (testNG.xml). The execution order is the one from the suite file, but when I look into the testng-result.xml and then in emailable-report.html, the tests are in a random order.

I would like the tests from the reports to be in the order that they have executed. Can you please help me in solving this issue?

Thank you

[ruslan-simonenko]

Hello,

I've created a pull request that, I believe, fixes this issue:
#554