Possibility to add priority on a class in TestNG xml #1324

slipwalker · 2017-02-06T14:29:13Z

Following by #1309 it would be helpful to have possibility to add priority on a class in TestNG xml file in case we have several classes within <classes></classes> tag. What I mean:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="My Suite" verbose="2"  parallel="tests" thread-count="10">

    <test name="Some Workflow Tests" parallel="none">
        <classes>
            <class name="com.somepackage.test.MyTests1" priority="1"/>
            <class name="com.somepackage.test.MyTests2" priority="2"/>
        </classes>
    </test>

</suite>

It's would be some kind of improvement and could help to avoid appearing similar issues (#1309) in the future. Is it real/doable to do like that?

The text was updated successfully, but these errors were encountered:

juherr · 2017-02-06T15:20:36Z

Everything in an annotation is supposed to be doable in xml too.

So, the request seems legit IMO ;)

juherr · 2017-02-06T15:21:06Z

@krmahadevan This request could be easy if you want to have a look on it ;)

krmahadevan · 2017-02-08T02:54:04Z

@juherr - I will take a look at this one. I was away for a couple of days due to a personal emergency.

krmahadevan · 2017-02-10T03:11:48Z

@slipwalker

If you remove the priority attribute from your @Test method, then your test classes will run in the order in which you specify them. Please see details below. Is that what you are expecting ?

Scenario : Test classes have one or more `@Test` methods defined with priorities

import org.testng.annotations.Test;

public class MyTests1 {

    @Test(priority = 1)
    public void test1() {
        System.out.println("test1 from " + getClass().getSimpleName() + " class");
    }

    @Test(priority = 2)
    public void test2() {
        System.out.println("test2 from " + getClass().getSimpleName() + " class");
    }
}

public class MyTests2 {

    @Test(priority = 1)
    public void test3() {
        System.out.println("test3 from " + getClass().getSimpleName() + " class");
    }

    @Test(priority = 2)
    public void test4() {
        System.out.println("test4 from " + getClass().getSimpleName() + " class");
    }
}

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="My Suite" verbose="2"  parallel="tests" thread-count="10">
    <test name="Some Workflow Tests" parallel="none" >
        <classes>
            <class name="org.rationale.emotions.github.issue1324.MyTests1"/>
            <class name="org.rationale.emotions.github.issue1324.MyTests2"/>
        </classes>
    </test>
</suite>

The output

[TestNG] Running:
  /Users/krmahadevan/playground/scrappable-project/src/test/resources/1324.xml
[ThreadUtil] Starting executor timeOut:2147483647ms workers:1 threadPoolSize:10
test1 from MyTests1 class
test3 from MyTests2 class
test2 from MyTests1 class
test4 from MyTests2 class
PASSED: test1
PASSED: test3
PASSED: test2
PASSED: test4

===============================================
    Some Workflow Tests
    Tests run: 4, Failures: 0, Skips: 0
===============================================

Scenario : Test classes have one or more `@Test` methods defined, without priorities

import org.testng.annotations.Test;

public class MyTests1 {

    @Test
    public void test1() {
        System.out.println("test1 from " + getClass().getSimpleName() + " class");
    }

    @Test
    public void test2() {
        System.out.println("test2 from " + getClass().getSimpleName() + " class");
    }
}

import org.testng.annotations.Test;

public class MyTests2 {

    @Test
    public void test3() {
        System.out.println("test3 from " + getClass().getSimpleName() + " class");
    }

    @Test
    public void test4() {
        System.out.println("test4 from " + getClass().getSimpleName() + " class");
    }
}

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="My Suite" verbose="2"  parallel="tests" thread-count="10">
    <test name="Some Workflow Tests" parallel="none">
        <classes>
            <class name="org.rationale.emotions.github.issue1324.MyTests1"/>
            <class name="org.rationale.emotions.github.issue1324.MyTests2"/>
        </classes>
    </test>
</suite>

[TestNG] Running:
  /Users/krmahadevan/playground/scrappable-project/src/test/resources/1324.xml
[ThreadUtil] Starting executor timeOut:2147483647ms workers:1 threadPoolSize:10
test1 from MyTests1 class
test2 from MyTests1 class
test3 from MyTests2 class
test4 from MyTests2 class
PASSED: test1
PASSED: test2
PASSED: test3
PASSED: test4

===============================================
    Some Workflow Tests
    Tests run: 4, Failures: 0, Skips: 0
===============================================

krmahadevan · 2017-02-10T03:17:34Z

@juherr - I would like to understand the expectations of this fix. Currently when parallel=none (parallelism is disabled), TestNG resorts to preserving order of the test classes. But the only time when this does not happen is when the test classes have one or more Test methods that contain priority. So how should we approach this fix ?

Should we change TestNG to start honouring the newly proposed priority attribute in the xml first and then followed by the priority attribute of the @Test method ?

i.e., consider the below example

import org.testng.annotations.Test;

public class MyTests1 {

    @Test(priority = 1)
    public void test1() {
        System.out.println("test1 from " + getClass().getSimpleName() + " class");
    }

    @Test(priority = 2)
    public void test2() {
        System.out.println("test2 from " + getClass().getSimpleName() + " class");
    }
}

public class MyTests2 {

    @Test(priority = 1)
    public void test3() {
        System.out.println("test3 from " + getClass().getSimpleName() + " class");
    }

    @Test(priority = 2)
    public void test4() {
        System.out.println("test4 from " + getClass().getSimpleName() + " class");
    }
}

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="My Suite" verbose="2"  parallel="tests" thread-count="10">
    <test name="Some Workflow Tests" parallel="none" preserve-order="true">
        <classes>
            <class name="org.rationale.emotions.github.issue1324.MyTests1" priority="2"/>
            <class name="org.rationale.emotions.github.issue1324.MyTests2" priority="1"/>
        </classes>
    </test>
</suite>

So we should execute them as below ?

test3 from MyTests2 class
test4 from MyTests2 class
test1 from MyTests1 class
test2 from MyTests1 class

Would appreciate if you could please help shed more light around this.

Also what should be the execution order for an xml such as the one below ?

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="My Suite" verbose="2"  parallel="tests" thread-count="10">
    <test name="Some Workflow Tests" parallel="none" preserve-order="true">
        <classes>
            <class name="org.rationale.emotions.github.issue1324.MyTests1" priority="1"/>
<!--Here SomeOtherTest is not having any priority in the xml. So what should its default priority be-->
            <class name="org.rationale.emotions.github.issue1324.SomeOtherTest"/>
            <class name="org.rationale.emotions.github.issue1324.MyTests2" priority="2"/>
        </classes>
    </test>
</suite>

Also should priority attribute of the <class> tag be honoured when parallel=methods|tests|classes|instances ? (Am guessing it shouldn't be, but still want to clarify)

juherr · 2017-02-10T09:48:07Z

In fact, I understand the need to override compiled things, like priority, at runtime.

So, my first idea was to add priority in XML and override Java configuration, where:
<class name="..." priority="2"/> == @Test(priority = 2) on a class (and overrides the value if one exists)
<include name="..." priority="2"/> == @Test(priority = 2) on a a method (and overrides the value if one exists)

Another idea (which can complete or replace the first one) is using XML, which is already supposed to be a way to order tests:

<test name="Some Workflow Tests" parallel="none" preserve-order="true">
        <classes>
            <class name="org.rationale.emotions.github.issue1324.MyTests1">
                <include name="test1"/>
            </class>
            <class name="org.rationale.emotions.github.issue1324.MyTests2">
                <include name="test4"/>
            </class>
            <class name="org.rationale.emotions.github.issue1324.MyTests1">
                <include name="test2"/>
            </class>
            <class name="org.rationale.emotions.github.issue1324.MyTests2">
                <include name="test3"/>
            </class>
        </classes>
    </test>

Expected output: test1, test4, test2, test3.

The current implement won't work because of priority is more important than preserve-order.
And I don't know yet if it is working without priority in tests (should be tested).
A naive change could be to reverse preserve-order and priority but preserve-order will have to be false by default then => could make regressions.

@cbeust WDYT?

cbeust · 2017-02-10T17:38:27Z

Intuitively, I think I'd expect priority to be above preserve-order. Any test failure if you make it so?

juherr · 2017-02-10T17:47:26Z

What do you mean by:

priority to be above preserve-order

Currently, priority is only available with @Test and preserve-order is only available in xml.
If priority is more important than preserver-order then it won't be possible to override order with xml.
if priority is less important than preserve-order then the default value must change to false otherwise it will break priority when xml is used (maybe more, but the tests should highlight it quickly).

Another option is having priority value in xml which will override values from @Test.

Both options, or only one, could be implemented.

krmahadevan · 2017-02-11T03:11:18Z

@juherr and @cbeust - I will wait to hear a conclusion on this one before fixing this :)

juherr · 2017-02-12T10:28:51Z

@cbeust What do you prefer?

add priority in xml?
set preserve-order=false by default and change importance between preserve-order and priority, aka "I know what I'm doing with my xml order"
Both
None
Something else

cbeust · 2017-02-13T06:52:55Z

might break existing users.

In doubt, and since this problem has been pretty rare so far, I'm inclined to do nothing.

juherr · 2017-02-13T08:44:51Z

ok

irhake · 2017-05-04T14:19:34Z

         <class name="com.versionone.tests.regression.Dimmer_Regression"  />
         
       
         <class name="com.versionone.tests.basetest.Nexia_DriverOFF" />
 </classes>

Hi Everyone,
Here am facing the issue as the class mentioned in the 2nd "com.versionone.tests.regression.Dimmer_Regression" is getting executed 1st
Is there anyway so that i can preserve the order.

krmahadevan · 2017-09-19T07:29:57Z

@goelgautam1994 - I would suggest that you submit a new issue, with some test code that can be used to recreate the problem.

nsteveson · 2017-11-01T18:18:04Z

Hello,

What Juherr suggested in # 1 above about putting priority into xml would be of enormous benefit in situations that I have found my company in. We have some selenium tests that can not run in parallel, and are functional different tests. As such our manager does not like the idea of putting them all into one class and using the @test priority tag. Here is a small snippet of our main test suite. The idea for us being able to use priority in the xml is to instruct testng to run one of the non parallel tests as soon as a thread is available. Currently without having priority in code usually see's these tests as the last running test, and if they got unlucky to get a freed up thread, they could take up to 30 minutes to finish up as the only running tests of the 10 parallel.

<suite name="Regression" parallel="tests" thread-count="10">
	<listeners>
		<listener class-name="admin.RetryListener" />
	</listeners>
	<test name="CompileTest">
		<classes>
			<class name="admin.CompileTest" />
		</classes>
	</test>
	  <!-- +++++++++++++++++++++++++++++Invoicing++++++++++++++++++++++++++++ -->
   <test name="TaxRulesInvoices">
    <classes>
      <class name="tests.Invoicing.RedraftInvoice" priority="1"/>
      <class name="tests.Invoicing.CancelInvoice" priority="1"/>
      <class name="tests.Invoicing.CreateInvoiceCancelTaxesAndDiscounts" priority="1"/>
    </classes>
  </test>
   <test name="USSalesTaxInvoices">
    <classes>
      <class name="tests.Invoicing.GenerateCSVExtract" priority="1"/>
      <class name="tests.Invoicing.ApproveInvoice" priority="1"/>
    </classes>
  </test>
  <test name="VATInvoices">
    <classes>
      <class name="tests.Invoicing.RejectInvoice " priority="1"/>
      <class name="tests.Invoicing.GeneratePDFExtract" priority="1"/>
    </classes>
  </test>
  <!-- ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -->
	<!--+++++++++++++++++++++++++++++++++++++++++++++ Assignment +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -->
	<test name="AmendAssignmentTest">
		<classes>
			<class name="tests.Assignments.AmendAssignmentWF3" />
		</classes>
	</test>
	<test name="ApproveHeadCountTrackingAssignmentTest">
		<classes>
			<class name="tests.Assignments.ApproveHeadCountTrackingAssignment" />
		</classes>
	</test>

This example is one of many my company runs into with concurrence issues where we see our regression tests only running 1 or 2 threads out of 10 for up to 30 minutes after the other several hundred tests have passed. Is adding priority to the xml out of the question?

dwightdhooge · 2018-03-02T16:43:17Z

@goelgautam1994 Did you fix your problem or created an issue for it ?

ankitagarwal021 · 2018-07-25T06:44:33Z

Class level priority is throwing error "Attribute priority is not allowed here"
<class name="tests.Invoicing.RejectInvoice " priority="1"/>
testng version 6.13.1

Please advice what need to be done for this?

krmahadevan · 2018-07-25T06:57:18Z

This was never implemented (Notice how the bug was closed as "Wont fix"). So there's no fix for this.

…

reemav · 2019-08-26T11:29:11Z

Is there any way to change priority runtime? @cbeust

krmahadevan · 2019-08-27T02:38:55Z

@reemav - Please make use of an IAnnotationTransformer to do this.

reemav · 2019-08-27T05:55:48Z

@krmahadevan , could you please provide an example of doing it with priority and how runtime I will be able to change the priority ? I saw most of the examples on Dataprovider .

Rameshwar-Juptimath · 2020-11-12T17:59:47Z

This was good feature to have. Not sure why this wasn't implemented.

AbhiCodeWorld · 2021-02-27T05:17:03Z

Yes, it would be a really great feature to have, adding priority to classes in XML.
And it should supersede the priority at the method level.
High priority class executes first -> then methods under it execute with method priority and so on.
Total control over execution... ✌(user feeling)

juherr added Feature: priority Type: Feature request xml labels Feb 6, 2017

juherr added the Won't fix label Feb 13, 2017

juherr closed this as completed Feb 13, 2017

ankitagarwal021 mentioned this issue Jul 25, 2018

Possibility to add priority on a class in TestNG xml #1875

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Possibility to add priority on a class in TestNG xml #1324

Possibility to add priority on a class in TestNG xml #1324

slipwalker commented Feb 6, 2017 •

edited

juherr commented Feb 6, 2017

juherr commented Feb 6, 2017

krmahadevan commented Feb 8, 2017

krmahadevan commented Feb 10, 2017

krmahadevan commented Feb 10, 2017 •

edited by juherr

juherr commented Feb 10, 2017

cbeust commented Feb 10, 2017

juherr commented Feb 10, 2017

krmahadevan commented Feb 11, 2017

juherr commented Feb 12, 2017

cbeust commented Feb 13, 2017

juherr commented Feb 13, 2017

irhake commented May 4, 2017

krmahadevan commented Sep 19, 2017 •

edited

nsteveson commented Nov 1, 2017 •

edited

dwightdhooge commented Mar 2, 2018

ankitagarwal021 commented Jul 25, 2018 •

edited

krmahadevan commented Jul 25, 2018 via email

reemav commented Aug 26, 2019 •

edited

krmahadevan commented Aug 27, 2019

reemav commented Aug 27, 2019

Rameshwar-Juptimath commented Nov 12, 2020

AbhiCodeWorld commented Feb 27, 2021 •

edited

Possibility to add priority on a class in TestNG xml #1324

Possibility to add priority on a class in TestNG xml #1324

Comments

slipwalker commented Feb 6, 2017 • edited

juherr commented Feb 6, 2017

juherr commented Feb 6, 2017

krmahadevan commented Feb 8, 2017

krmahadevan commented Feb 10, 2017

Scenario : Test classes have one or more @Test methods defined with priorities

Scenario : Test classes have one or more @Test methods defined, without priorities

krmahadevan commented Feb 10, 2017 • edited by juherr

juherr commented Feb 10, 2017

cbeust commented Feb 10, 2017

juherr commented Feb 10, 2017

krmahadevan commented Feb 11, 2017

juherr commented Feb 12, 2017

cbeust commented Feb 13, 2017

juherr commented Feb 13, 2017

irhake commented May 4, 2017

krmahadevan commented Sep 19, 2017 • edited

nsteveson commented Nov 1, 2017 • edited

dwightdhooge commented Mar 2, 2018

ankitagarwal021 commented Jul 25, 2018 • edited

krmahadevan commented Jul 25, 2018 via email

reemav commented Aug 26, 2019 • edited

krmahadevan commented Aug 27, 2019

reemav commented Aug 27, 2019

Rameshwar-Juptimath commented Nov 12, 2020

AbhiCodeWorld commented Feb 27, 2021 • edited

slipwalker commented Feb 6, 2017 •

edited

Scenario : Test classes have one or more `@Test` methods defined with priorities

Scenario : Test classes have one or more `@Test` methods defined, without priorities

krmahadevan commented Feb 10, 2017 •

edited by juherr

krmahadevan commented Sep 19, 2017 •

edited

nsteveson commented Nov 1, 2017 •

edited

ankitagarwal021 commented Jul 25, 2018 •

edited

reemav commented Aug 26, 2019 •

edited

AbhiCodeWorld commented Feb 27, 2021 •

edited