Serenity BDD: Html report is having more number of tests than expected

[chanduguthikonda]

I recently upgraded my Serenity version from 3.2.5 to 4.0.29 and observing the difference in the number of tests executed in the HTML report randomly.
Example:
Actual number of scenarios executed: 6
Actual number of tests executed (including all examples): 12

Number of scenarios reported: 6
Number of test cases reported: 17

Number of scenarios are being reported correctly but the test case numbers are not in sync.
Observing this behavior randomly when there are Pending test cases reported in the HTML report.
I'm observing inconsistent behavior in the report for the same set of tests in multiple runs.
I'm also deleting the target folder before each run

Below is my build.gradle:

`defaultTasks 'clean', 'test', 'aggregate'

repositories {
mavenLocal()
jcenter()
mavenCentral()
}
buildscript {
repositories {
mavenLocal()
jcenter()
maven {
url "https://plugins.gradle.org/m2/" }
}
dependencies {
classpath("net.serenity-bdd:serenity-gradle-plugin:4.0.27")
classpath("net.serenity-bdd:serenity-jira-plugin:1.12.0")
classpath("net.serenity-bdd:serenity-jira-requirements-provider:1.12.0")
}
}

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'net.serenity-bdd.serenity-gradle-plugin'
//apply plugin: 'net.serenity-bdd.aggregator'

sourceCompatibility = 1.9
targetCompatibility = 1.9

ext {
serenityVersion = '4.0.29'
junitVersion = '4.12'
assertJVersion = '3.8.0'
logbackVersion = '1.2.3'
}

dependencies {
implementation 'net.serenity-bdd:serenity-core:4.0.29'
implementation 'net.serenity-bdd:serenity-cucumber:4.0.29'
implementation 'net.serenity-bdd:serenity-screenplay:4.0.29'
implementation 'net.serenity-bdd:serenity-screenplay-webdriver:4.0.29'
implementation 'net.serenity-bdd:serenity-rest-assured:4.0.29'
implementation 'net.serenity-bdd:serenity-screenplay-rest:4.0.29'
implementation 'junit:junit:4.12'
implementation 'org.assertj:assertj-core:3.8.0'
implementation 'com.oracle.database.jdbc:ojdbc11:21.7.0.0'
testImplementation 'com.h2database:h2:2.1.210'
implementation group: 'au.com.bytecode', name: 'opencsv', version: '2.4'
implementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.12.0'
implementation group: 'io.cucumber', name: 'datatable', version: '1.0.3'
implementation group: 'org.apache.poi', name: 'poi', version: '4.1.2'
implementation group: 'org.yaml', name: 'snakeyaml', version: '1.29'
implementation group: 'org.apache.poi', name: 'poi-ooxml', version: '4.1.2'
implementation 'com.googlecode.json-simple:json-simple:1.1.1'
implementation 'net.serenity-bdd:serenity-jira-requirements-provider:1.12.0'
implementation group: 'com.microsoft.ews-java-api', name: 'ews-java-api', version: '2.0'
testImplementation 'ch.qos.logback:logback-classic:1.4.14'
testImplementation 'org.projectlombok:lombok:1.18.26'
compileOnly 'org.projectlombok:lombok:1.18.26'
annotationProcessor 'org.projectlombok:lombok:1.18.26'
testAnnotationProcessor 'org.projectlombok:lombok:1.18.26'
implementation fileTree(dir: 'lib', include: ['*.jar'])
implementation 'org.springframework:spring-jdbc:5.1.10.RELEASE'
implementation 'org.springframework:org.springframework.transaction:3.1.4.RELEASE'
implementation group: 'javax.xml.ws', name: 'jaxws-api', version: '2.2.9'
implementation 'org.jsoup:jsoup:1.16.1'
implementation 'javax.xml.bind:jaxb-api:2.3.1'
implementation group: 'org.apache.pdfbox', name: 'pdfbox', version: '2.0.8'
implementation 'org.bouncycastle:bcprov-jdk15on:1.68'
implementation 'org.bouncycastle:bcpkix-jdk18on:1.77'
}
test {
testLogging.showStandardStreams = true
systemProperties System.getProperties()
maxParallelForks 8
}
gradle.startParameter.continueOnFailure = true
test.finalizedBy(aggregate)`

Appreciate any pointers. Thank you

[wakaleo]

When you say "randomly", do you mean the results change each time you run the tests?

[chanduguthikonda]

@wakaleo Thanks for your quick response.
Not for every run but the behavior is inconsistent across multiple runs with the same set of test cases. I've added the screenshots for your reference.

[wakaleo]

There is a known issue related to the test counts in the presence of failing tests that we are looking into.

[chanduguthikonda]

@wakaleo Is this test counts issue specific to the latest version 4.0.29, or does it persist across all versions with a major update in the 4.x.x series?

[wakaleo]

Try with 4.0.43

[chanduguthikonda]

@wakaleo still seeing the same behavior with 4.0.43. Reporting more tests than expected in the HTML report.

[chanduguthikonda]

@wakaleo Please let me know if the issue is fixed in the latest versions.
Thank you

[wakaleo]

Can you try with version 4.1.0? It should be fixed in this version. If it is not, please provide a sample project that reproduces the issue with version 4.1.0.

[chanduguthikonda]

Thanks for the update @wakaleo, Let me try with 4.1.0.