Run scenarios in parallel with Gradle

[iholub]

I have 50 scenarios in my cucumber feature file.
Can I run them in parallel using Gradle? (so that for example 5 browsers will be used)
If yes, is there any doc for this?
If no, is it possible with maven?

[prashantigs]

I want have similar question i have cucumber bdd, junit, gradle and i want to run test cases parallel

[wakaleo]

Parallel execution in Cucumber isn't supported yet. It will be in a future version.

[prashantigs]

Thanks wakaleo, currently i am using cucumber-java version: 6.11.0

Please let me know once we have parallel execution for cucumber. Thanks in advance

[wakaleo]

Parallel execution is in any case only supported in the most recent versions of Cucumber - I don't think it is supported in Cucumber 6.

[prashantigs]

In serenity official document it says cucumber 6 supports parallel execution. will it help If we switch the project back to Maven?

[wakaleo]

In serenity official document it says cucumber 6 supports parallel execution. will it help If we switch the project back to Maven?

There were/are a lot of issues with parallel execution in Cucumber.

[prashantigs]

Do you think if we change Gradle to maven will help? i am unable to find proper community discussion on gradle with cucumber parallel execution.

[zzoubian]

This is hit and miss. You can try this:

1. In your build.gradle you can add a test task that contains the following:
   maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1
   and manually split your features into separate runners.
2. gradle test --max-workers=desired number of works --parallel

If you are using gitlab ci or jenkins you can split your tests into a batch of scenarios. For example in gitlab ci, you would do something like the following as a script:
gradle -Dserenity.batch.count=$CI_NODE_TOTAL -Dserenity.batch.count=$CI_NODE_INDEX test

Finally, if you are adventurous, you can use cucumber cli, and set the number of threads to use. Again, you can setup a task like follows:
task runTestsParallel (type: JavaExec) {
dependsOn compileTestJava
classpath = sourceSets.main.runtimeClasspath + sourceSets.test.runtimeClasspath
mainClass = "com.whatever.Main"
args = ['--threads', '4', '--glue', 'com.whatever.stepdefinitions', 'src/test/resources/features']
}

Here is an example of the Main class you can use:

public class Main {
  
    public static void main(String[] argv) throws Throwable {
        Supplier<ClassLoader> classLoaderSupplier = ClassLoaders::getDefaultClassLoader;
        run(argv, classLoaderSupplier);
    }
    public static byte run(String[] argv, Supplier<ClassLoader> classLoaderSupplier) throws IOException {
        RuntimeOptions runtimeOptions = new CommandlineOptionsParser(System.out).parse(argv)
                .addDefaultGlueIfAbsent()
                .addDefaultFeaturePathIfAbsent()
                .addDefaultSummaryPrinterIfAbsent()
                .build();
        CucumberWithSerenity.setRuntimeOptions(runtimeOptions);
        Runtime runtime = CucumberWithSerenityRuntime.using(classLoaderSupplier, runtimeOptions);
        runtime.run();
        return runtime.exitStatus();
    }
}

[prashantigs]

Thanks i will try this and update the status.

[prashantigs]

@zzoubian Is there any step by step tutorial or any blog that explains the above solutions in detail? can you please provide the link. Thanks

[GianniGiglio]

@prashantigs The most straitforward way of running features files in parallel is using gradle's test configuration and forks.
I have an example project https://github.com/GianniGiglio/serenity-cucumber-threaded
See https://github.com/GianniGiglio/serenity-cucumber-threaded/blob/master/build.gradle#L47 for the details.

The above solution using main has a different behavior. This will run the scenario's in parallel rather than the features in parallel. This solution results in a overall faster execution of the tests since all scenarios are distributed among threads. I have a similar solution implemented however it's not fault tolerated. From time to time test are failing. This is probably caused by the class BaseStepListner not being fully threadSafe (fyi @wakaleo ).

[prashantigs]

Hello @GianniGiglio thanks for the solution I have tested your project it's working as expected.

I have a questions
In the 2 runner files, you had mentioned the path of two different feature file package i.e /search/keyword and /search/banana

Question1: Suppose if I have 100 package under /search package I have to create 100 runner file?
Question2: is it possible to use one runner file with a feature file root folder path? i tried this way it's not executing tests in parallel
example:
package starter;

import io.cucumber.junit.CucumberOptions;
import net.serenitybdd.cucumber.CucumberWithSerenity;
import org.junit.runner.RunWith;

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(
plugin = {"pretty"},
features = "src/test/resources/features/search/"
)
public class CucumberTestSuite {}

[prashantigs]

@GianniGiglio i can able to execute your code parallelly however when I try my project it's still executing test cases in sequential mode. i am suspecting two things

1). my framework structure is wrong - Do i need to maintain the same framework structure as yours?
2). I think my gradle file dependencies are wrong. here it is

defaultTasks 'clean', 'test', 'aggregate'
apply plugin: 'java'
apply plugin: 'net.serenity-bdd.aggregator'

group 'org.automationserenity'
version '1.0-SNAPSHOT'

repositories {
mavenCentral()
}

buildscript {
project.ext {
serenityVersion = '2.3.10'
}
repositories {
mavenLocal()
jcenter()
}
dependencies {
classpath('net.serenity-bdd:serenity-gradle-plugin:' + serenityVersion)

}

}

sourceCompatibility = 1.8
targetCompatibility = 1.8

ext {
serenityCoreVersion = '3.1.1'
junitVersion = '4.11'
assertJVersion = '3.21.0'
logbackVersion = '1.2.3'
serenityCucumberVersion = '2.6.0'
}

dependencies {

testImplementation "net.serenity-bdd:serenity-core:${serenityCoreVersion}",
        "net.serenity-bdd:serenity-junit:${serenityCoreVersion}",
        "net.serenity-bdd:serenity-screenplay:${serenityCoreVersion}",
          "net.serenity-bdd:serenity-cucumber6:${serenityCucumberVersion}",
        "junit:junit:${junitVersion}",
        "org.assertj:assertj-core:${assertJVersion}"
implementation 'org.junit.jupiter:junit-jupiter:5.7.0'
implementation group: 'io.cucumber', name: 'cucumber-java', version: '6.11.0'
implementation group: 'org.apache.httpcomponents', name: 'httpclient', version: '4.5.13'
implementation group: 'org.apache.httpcomponents', name: 'httpcore', version: '4.4.14'
implementation group: 'org.json', name: 'json', version: '20210307'
implementation group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.12.3'
implementation 'com.github.javafaker:javafaker:1.0.2'
implementation group: 'org.apache.poi', name: 'poi', version: '5.0.0'
testImplementation group: 'com.jayway.restassured', name: 'rest-assured', version: '2.9.0'
testImplementation group: 'io.rest-assured', name: 'rest-assured', version: '3.2.0'
implementation group: 'io.github.bonigarcia', name: 'webdrivermanager', version: '3.0.0'
implementation group: 'org.jboss.aerogear', name: 'aerogear-otp-java', version: '1.0.0'

// implementation group: 'org.seleniumhq.selenium', name: 'selenium-java', version: '4.0.0-beta-4'
implementation group: 'commons-configuration', name: 'commons-configuration', version: '1.10'
implementation group: 'net.serenity-bdd', name: 'serenity-reports', version: '2.4.51'
// https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml
implementation group: 'org.apache.poi', name: 'poi-ooxml', version: '5.0.0'
// https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple
implementation group: 'com.googlecode.json-simple', name: 'json-simple', version: '1.1'
implementation group: 'net.serenity-bdd', name: 'serenity-zalenium', version: '2.3.13'
// https://mvnrepository.com/artifact/io.appium/java-client
implementation group: 'com.applitools', name: 'eyes-selenium-java3', version: '3.208.2'
implementation group: 'com.jayway.jsonpath', name: 'json-path', version: '2.6.0'
implementation 'org.mongodb:mongodb-driver:3.12.10'
implementation group: 'org.mongodb', name: 'mongo-java-driver', version: '3.12.8'
implementation 'org.mongodb:mongodb-driver-sync:4.5.0'
// https://mvnrepository.com/artifact/com.github.ozlerhakan/poiji
implementation group: 'com.github.ozlerhakan', name: 'poiji', version: '3.1.1'

}

test {
testLogging.showStandardStreams = true
maxParallelForks = 2
systemProperties System.getProperties()
}

tasks.withType(Test) {

filter {

    includeTestsMatching "*AccountsTestRunner"

}

}

gradle.startParameter.continueOnFailure = true
test.finalizedBy(aggregate, reports)
apply plugin: 'maven-publish'

Can some one help me on this?

[wakaleo]

Parallel execution of Cucumber scenarios isn't supported yet.

[prashantigs]

in the cucumber official website mentioned, we can achieve parallel execution of cucumber scenarios. https://cucumber.io/docs/guides/parallel-execution/

Not sure which one is correct. please let me know if there are any issues with the serenity, cucumber, JUnit, and gradle project so that we can find alternative solutions as we have 500+ test cases automated with serenity. Its night mare for us to execute in sequential order

[wakaleo]

Cucumber parallel execution of scenarios with Serenity is not currently supported, though it will work at a feature level. Cucumber parallel execution with Maven and Gradle does not always work as documented, which is a Cucumber issue.

[prashantigs]

Hello Wakaleo, is it possible to run test cases parallelly using the below options? with serenity cucumber?

1). Running Serenity tests on a Selenium Grid and 2). Running Serenity tests in parallel batches

above 2 options mentioned in serenity official document https://serenity-bdd.github.io/theserenitybook/latest/serenity-parallel-batches.html

[Dsilva-Rohan]

Need help and proper guidance on how to run parallel execution on Gradle and Cucumber with selenium, I have searched number of blogs there are no any proper guidance or exact solution, can any one help me in this? but there are good suggestion with Maven but not with gradle

[prashantrbs]

I face the same problem so i converted entire project to maven and now i can able to execute my tests in parallel mode

[Dsilva-Rohan]

@prashantrbs any idea how we can achieve this with Gradle and JUnit parallel execution ? for me at least feature file parallel execution also fine. Because in maven I know that we can use plugin section and we can do with Test-NG and Junit both for scenario level and Feature level (Junit) in POM.XML....... but I need it in gradle only.......

[prashantrbs]

Even i had same issue in gradle so i searched #'s blogs unfortunately that did not help so change entire project to maven

[Dsilva-Rohan]

@wakaleo, Why we have lot of challenges with Selenium cucumber with gradle to execute the parallel execution? but same case is not applicable for Maven with selenium cucumber. At least in maven there are 2 ( Failesafe and Surefire) plugins are available to configure with Junit (feature level ) parallel execution and Test-NG( scenario level) execution. But with gradle there are no any tutorial or community support for the parallel execution even we do not have good blogs or video. Can you please help me here....... we are waiting for this from very long time.

[wakaleo]

@wakaleo, Why we have lot of challenges with Selenium cucumber with gradle to execute the parallel execution? but same case is not applicable for Maven with selenium cucumber. At least in maven there are 2 ( Failesafe and Surefire) plugins are available to configure with Junit (feature level ) parallel execution and Test-NG( scenario level) execution. But with gradle there are no any tutorial or community support for the parallel execution even we do not have good blogs or video. Can you please help me here....... we are waiting for this from very long time.

Serenity BDD is an Open Source project. So if this feature is valuable for your company, you have a few options:
(a) you can wait for someone to pick it up and work on it for free, in their own personal time. (Note that there is no SLA or guarantee that any particular feature request will be implemented of course).
(b) you can do it yourself, or pay someone to do it
(c) you can become a sponsor of the Serenity BDD project, and thus have a say in what features are prioritised (and feel good about supporting the open source development community).

[GianniGiglio]

@prashantigs @Dsilva-Rohan you can find an example using gradle here https://github.com/GianniGiglio/serenity-cucumber-threaded
This will run feature files in parallel
If you want to run scenarios in parallel you can follow the example provided here https://github.com/serenity-bdd/serenity-core/tree/main/serenity-cucumber-smoke-tests that use junit5 to do the parallelization.

Tobe clear both solutions work but the junit 5 solution will yield the fastest execution since scenarios are run in parallel rather than feature files.