TestNG tests not running in parallel mode with gradle 4.5.1 if there is no parallel attribute inside the test tag

[pgsung]

TestNG tests not running in parallel mode with gradle 4.5.1 if there is no parallel attribute inside the test tag. Originally reported to TestNG issue testng-team/testng#1701, maintainer (@krmahadevan ) suggested file an issue with gradle.

This seems to be an issue with Gradle. I am not that well versed with Gradle. But my debugging seems to suggest that when we hit the flow here the parallel mode is getting reset back to null. This issue has been fixed in TestNG as of 6.14.2. You can try right clicking on your testng.xml suite file in your project and running it. You should see different thread ids.

Expected Behavior

Tests should run in the parallel mode with gradle 4.5.1 with testng.xml at https://github.com/pgsung/testng-bug/blob/master/src/test/resources/testng.xml
and source code at https://github.com/pgsung/testng-bug
test classes https://github.com/pgsung/testng-bug/tree/master/src/test/java/parallel/test
, thread ids should be different.

[RemoteTestNG] detected TestNG version 6.14.2
11
12
10

Current Behavior

Tests not running in parallel mode with gradle 4.5.1

content-backoffice-ci-uitest Suite > thread-test > parallel.test.Test1.runTests STANDARD_OUT
    10
Running test: Test method runTests(parallel.test.Test2)

content-backoffice-ci-uitest Suite > thread-test > parallel.test.Test2.runTests STANDARD_OUT
    10
Running test: Test method runTests(parallel.test.Test3)

content-backoffice-ci-uitest Suite > thread-test > parallel.test.Test3.runTests STANDARD_OUT
    10

Context

Most selenium browser tests rely on gradle TestNG runner to run parallel tests with selenium grid. The current workaround is to add parallel="classes" into test tag in textng.xml or downgrade the TestNG to 6.11.

Steps to Reproduce (for bugs)

1. clone the repo at https://github.com/pgsung/testng-bug
2. run the test with command ./gradlew clean test

Your Environment

• Build scan URL:
  https://scans.gradle.com/s/ygiuoh25qh3i4

[oehme]

@pgsung You mention Gradle 4.5.1 in the title, is this a new issue? Did it work with some previous version of Gradle?

[blindpirate]

Thanks for reporting and the reproducible case. I've confirmed this is on Gradle's side. I'll try to fix it soon.

[pgsung]

@oehme should be reproducible with Gradle 4.3.1 as well so it is not really a new issue.

[kool79]

Rootcause of the issue is how gradle initialize testng. Gradle always overrides values of options threadCount and parallel even if they are not explicitely set in gradle file (see here)
If you did not call setParallel() and setThreadCount() then gradle will force threadCount to 1 and parallel to null (default values in TestNGOptions). In turns testNg converts 'null' for threadCount to XmlSuite.ParallelMode.NONE. Also it overrides both those values for suite (<suite> tag)
To fix this gradle must not call

        testNg.setParallel(options.getParallel());
        testNg.setThreadCount(options.getThreadCount());

if they were not initialized in gradle file.