Serenity test reports limited to 20 test cases when build is failing

[aybartsch]

Hi Serenity Developers.

Since the update to version 3.6.12 I'm always facing incomplete serenity reports when one or more tests are failing. For example:

38 tests completed, 2 failed
> Task :runDevelopmentTests FAILED
> Task :stopDockerCompose SKIPPED
> Task :reports
Generating Additional Serenity Reports for e2e to directory /Users/andrebartsch/IdeaProjects/nvz-clean/e2e/target/site/serenity
  - Single Page HTML Summary: file:///Users/andrebartsch/IdeaProjects/nvz-clean/e2e/target/site/serenity/serenity-summary.html
  - JSON Summary: file:///Users/andrebartsch/IdeaProjects/nvz-clean/e2e/target/site/serenity/serenity-summary.json
> Task :aggregate
Generating Serenity Reports
  - Main report: file:///Users/andrebartsch/IdeaProjects/nvz-clean/e2e/target/site/serenity/index.html
      - Test Root: null
      - Requirements base directory: null
FAILURE: Build failed with an exception.

The above gradle build was running 38 tests with 2 tests failing but all reports (json, summary, main) contains 20 test cases only. There are no more than 20 test cases and it might happen that the failing one is missing in the report, too. Surprisingly, the file "target/site/serenity/results.csv" is containing all 38 tests. Do you have any idea?

Serenity general settings used for reporting:

serenity.console.colors = false
serenity.driver.capabilities = custom-driver
serenity.logging = VERBOSE
serenity.project.name = NVZ E2E Testing
serenity.report.show.manual.tests = true
serenity.report.show.releases = true
serenity.report.show.step.details = true
serenity.requirement.types = feature, story
serenity.step.delay = 1500
serenity.store.html.source = true
serenity.take.screenshots = AFTER_EACH_STEP
serenity.test.root = com.westernacher.nvz.e2e.tests.serenity
serenity.use.single.browser = true
serenity.verbose.steps = true
serenity.jquery.integration = false
feature.file.language = de
feature.file.encoding = utf8
report.tagtypes = feature, story

Thanks and best regards, André

[aybartsch]

I was bumping the version of serenity to 3.7.1 and plugin to 3.7.0 but the problem still persists.

[aybartsch]

The last working version is 3.2.5. The problem was introduced with serenity gradle plugin version 3.3.0.

[wakaleo]

Does this work if you run with Maven?

[aybartsch]

Hi @wakaleo. We are mainly working with gradle, means maven knowledge is limited and would cause higher effort to switch and test based on maven. So, I can't tell you.

[wakaleo]

Hi @aybartsch I'm more familiar with Maven than Gradle. There are longstanding issues with Gradle due to the way it handles classpaths if I recall correctly which can cause problems with the reports. I don't have time to investigate currently, as I don't use Gradle very actively. However if you would like to investigate on your side, or get your company to sponsor some work in this area via a support contract, we should be able to get this working properly.

[aybartsch]

@wakaleo Okay, I will try to find time to retest this issue with maven build. Will come back to you...

[aybartsch]

The reports are no longer limited after updating to version 4.0.15.