Testcases out of order in XML file in junitreport folder when using testng

[gaoqiang0306]

TestNG Version 6.9.10 and 6.9.13.6

Note: only the latest version is supported

Expected behavior

testcase should be as the same order as the test cases run

Actual behavior

the order random each time and testng-results.xml was correct which cannot used for junit report when using ant and maven

Is the issue reproductible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Part of my testcases

    @Test(priority = 1)
    public void testRoles001_Post() {
       *******
    }

    @Test(priority = 2)
    public void testRoles002_Post() {
       *******
    }
    @Test(priority = 3)
    public void testRoles003_Post() {
       *******
    }
    @Test(priority = 4)
    public void testRoles004_Post() {
       *******
    }

but the junitreports was showed not in order as below

    <?xml version="1.0" encoding="UTF-8"?>
    <!-- Generated by org.testng.reporters.JUnitReportReporter -->
    <testsuite name="com.test.rest.testcase.common.RolesRestTest" tests="14" failures="0" timestamp="3 Dec 2016 16:13:35 GMT" time="38.868" errors="0">
       <testcase name="testRoles004_Post" time="2.596" classname="com...test.rest.testcase.common.RolesRestTest"/>
       <testcase name="testRoles002_Post" time="1.224" classname="com...test.rest.testcase.common.RolesRestTest"/>
       <testcase name="testRoles003_Post" time="0.764" classname="com...test.rest.testcase.common.RolesRestTest"/>
      <testcase name="testRoles001_Post" time="2.680" classname="com...test.rest.testcase.common.RolesRestTest"/>
    </testsuite> 

I want to use ant and XML files in junitreport folder to generate a html, if the order random each time, it
will be confused when the report were saw by others

testng report

[krmahadevan]

@gaoqiang0306 - Can you please help add some clarity around why you feel that the order of execution if different in the xml reports will confuse the users ? Why is the order so important here ? Just curious to hear this part because I understand JUnit doesn't have dependency of tests, and so order shouldn't really matter in reports.

[gaoqiang0306]

@krmahadevan At the beginning, we did UT with Junit 4 and @FixMethodOrder can be used which can be used like dependency , so the order for the report was like 001,002,003, 004. Recently, QA Manager team want us move to TestNG due to more powerful, but the order in XML was random, they maybe confused why. So in my opinion, the testcases order should be the same as testng-results.xml showed, follow the order we defined in our test cases. Thanks for your support.

[juherr]

I think the testng and junit reports should, at least, have the same result order.
@gaoqiang0306 Could you join the testng report of your sample too?

[gaoqiang0306]

@juherr Done, please check.

[juherr]

@krmahadevan Did you have a look on it? I think it could be a good opportunity for a XmlJUnitReporter2. WDYT?

[krmahadevan]

@juherr I haven't taken a look yet. I will take a look and update.

[krmahadevan]

@juherr - Why not fix this in the current JUnitReportReporter instead of creating another one ? If we are going to be trying to just fix the ordering of the tests listed in the xml, then we shouldn't be having any impact on the end users no ? Because I think xml consumption shouldn't be impacted based on the ordering of the nodes. Please let me know. Based on what we decide I will proceed further.

[juherr]

I had a quick look and I think the change may be as important as a rewrite. But maybe Im wrong too ;)

[krmahadevan]

@juherr - I was able to bring the ordering with just introducing a Comparator and resorting to sorting using Priority. I have the code ready at home as well. I will perhaps raise a PR so that it adds clarity to what I am talking of. I will raise the PR as soon as I go home. But all of this assumption is based on the fact that the end-user will not be impacted if the ordering of the testcase is changed when their test does not involve priorities (because my comparator is going to be sorting on priority)

[juherr]

I'm not sure a sort on priority will be enough. And the request is to have the same order than the testng report. (I didn't check how it is sorted there)

[krmahadevan]

@juherr - I don't think that the user is expecting a parity between the TestNG reports and the JUnit reports in terms of order. I think the user is basically expecting that the JUnit reports honor the priority (because that is what is essentially determining the execution order) and list out test cases in the same order of execution. Also looking at the screenshot, the emailable report (which is TestNG report) seems to be ordering the tests alphabetically (within each test class of-course) See here

The contents of testng-results.xml which is generated by XMLReporter resorts to doing chronological sorting (Please see here)

Since the user explicitly mentions the usage of FixMethodOrder in Junit world to enforce execution order, I am guessing that the user is resorting to getting the same experience via the usage of priority attribute and as such is now expecting that the JUnit reports have the same order as per the order of execution. This is why I felt that we just need to add a comparator that uses the priority for comparisons and sort the methods before creating the reports.

Please let me know if there's any gap here.

[juherr]

@krmahadevan In fact, I think all reporters should have the same order and here it is a good opportunity to have it for the JUnit reporter. Another option could be to have the same order than JUnit itself.

But don't worry, I'm just sharing my ideas and I won't block a pull request based on priority order ;)

[krmahadevan]

But don't worry, I'm just sharing my ideas and I won't block a pull request based on priority order ;)

@juherr - Please don't get me wrong. I was only trying to ensure that I am understanding the bug properly so that I can fix it correctly :)

In fact, I think all reporters should have the same order and here it is a good opportunity to have it for the JUnit reporter.

That sounds like a fair ask. I was thinking that maybe its time we provided some mechanism wherein the ordering of methods is something that our end users can provide to us as a comparator and we just use that to honor the sorting of methods (within the scope of a test class) based on their order. This I thought would be extremely useful for people who want to tweak the listing of methods in the default reports (over which our users currently don't have much of a control since they are auto-wired)

[gaoqiang0306]

@krmahadevan @juherr Thanks for your help, another question, how can i get the latest code using maven, which version i needed?

[krmahadevan]

@gaoqiang0306 - I am not sure if the snapshot versions of TestNG are being published anywhere.
@juherr @cbeust - Please advise.

[juherr]

Snapshots are available on https://oss.sonatype.org/content/repositories/snapshots/

[krmahadevan]

@juherr - Is this current snapshot version ?

I was thinking that the snapshot version would be something after 6.10 ?

[juherr]

@krmahadevan Yes, the version must be updated in gradle and kobalt files ;)

[krmahadevan]

ok. So I figured out that it should be changed here. Next question is.. what should be the new version ? :)

[juherr]

@krmahadevan 6.10.1 looks good

[juherr]

@krmahadevan And don't forget https://github.com/cbeust/testng/blob/master/kobalt/src/Build.kt#L18 if you want to update the value ;)

[cbeust]

Yes, 6.10.1 is a good default.

…

-- Cédric

On Tue, Dec 13, 2016 at 4:33 AM, Julien Herr ***@***.***> wrote: @krmahadevan <https://github.com/krmahadevan> And don't forget https://github.com/cbeust/testng/blob/master/kobalt/src/Build.kt#L18 if you want to update the value ;) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1262 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFoonSiQsTAgsdJSddS-wcxEbJONaTPks5rHpCDgaJpZM4LD0LD> .

[krmahadevan]

Thanks. I fixed this and sent a pull request #1274

[gaoqiang0306]

@krmahadevan @juherr @cbeust thanks, all of you guys. my issue was fixed so quickly, nice work!!!