Reports not getting generated when test cases cross certain number

[wee-jay]

We are using this library in our project for reporting and there is an issue that we found when our test cases went from 300 to 500 and above.
When I run the Runner file with a tag name having 50 to 100 test cases serenity reporting works fine and triggers the email, but when the test cases count went above a certain number like 500 (Now we have 700+) the report is not generated.
Scripts pause at the end waiting for serenity to generate the reports. Even after waiting for around 10 hours there was no luck.

Also I observed that it takes 30 min for just report generation for 200 to 300 test cases where the test cases get executed in 20 to 30 min, half of the time is consumed by serenity report generation :)

Can u please suggest us what could have gone wrong in the configuration? Or let us know if it is a genuine issue and ETA for fix.

We are using 3.1.15 version
Please note Pinaka is our internal library which is also using same serenity version.

Below are the list of dependencies from pom.xml used by our project.

<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<browser>chrome</browser>
		<app.env>test</app.env>
		<headless>true</headless>
		<env>rec</env>
		<serenity.version>3.1.15</serenity.version>
		<serenity.maven.version>3.1.15</serenity.maven.version>
		<serenity.cucumber.version>3.1.15</serenity.cucumber.version>
		<threadCount>2</threadCount>
		<java.version>1.8</java.version>
		<maven.compiler.version>3.2</maven.compiler.version>
		<pinaka.version>2.8</pinaka.version>
		<postgresql.version>42.2.16</postgresql.version>
	</properties>

<dependencies>
		<dependency>
			<groupId>com.cbre.pe</groupId>
			<artifactId>cbre-bdd-framework</artifactId>
			<version>${pinaka.version}</version>
		</dependency>
		<dependency>
			<groupId>net.minidev</groupId>
			<artifactId>json-smart</artifactId>
			<version>2.3</version>
			<exclusions>
				<exclusion>
					<groupId>org.json</groupId>
					<artifactId>json</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>com.beust</groupId>
			<artifactId>jcommander</artifactId>
			<version>1.78</version>
		</dependency>
		<dependency>
			<groupId>com.cbre.password.encrypter</groupId>
			<artifactId>CbrePasswordManager</artifactId>
			<version>0.0.1</version>
		</dependency>
		<dependency>
			<groupId>javax.mail</groupId>
			<artifactId>mail</artifactId>
			<version>1.4</version>
		</dependency>
		<!-- https://mvnrepository.com/artifact/javax.json/javax.json-api -->
		<dependency>
			<groupId>javax.json</groupId>
			<artifactId>javax.json-api</artifactId>
			<version>1.1.4</version>
		</dependency>
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-api</artifactId>
			<version>1.7.5</version>
		</dependency>
		<dependency>
			<groupId>com.jayway.jsonpath</groupId>
			<artifactId>json-path</artifactId>
			<version>2.4.0</version>
		</dependency>
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-log4j12</artifactId>
			<version>1.7.5</version>
		</dependency>
		<dependency>
			<groupId>net.serenity-bdd</groupId>
			<artifactId>serenity-core</artifactId>
			<version>${serenity.version}</version>
		</dependency>
		<dependency>
			<groupId>net.serenity-bdd</groupId>
			<artifactId>serenity-junit</artifactId>
			<version>${serenity.version}</version>
		</dependency>
		<dependency>
			<groupId>net.serenity-bdd</groupId>
			<artifactId>serenity-rest-assured</artifactId>
			<version>${serenity.version}</version>
		</dependency>
		<dependency>
			<groupId>net.serenity-bdd</groupId>
			<artifactId>serenity-cucumber</artifactId>
			<version>${serenity.cucumber.version}</version>
		</dependency>
		<dependency>
			<groupId>info.cukes</groupId>
			<artifactId>cucumber-java</artifactId>
			<version>1.2.5</version>
		</dependency>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>4.11</version>
		</dependency>
		<dependency>
			<groupId>com.fasterxml.jackson.dataformat</groupId>
			<artifactId>jackson-dataformat-xml</artifactId>
			<version>2.9.8</version>
		</dependency>
		<dependency>
			<groupId>xerces</groupId>
			<artifactId>xercesImpl</artifactId>
			<version>2.6.0</version>
		</dependency>
		<dependency>
			<groupId>com.opencsv</groupId>
			<artifactId>opencsv</artifactId>
			<version>3.3</version>
		</dependency>
		<dependency>
			<groupId>org.apache.poi</groupId>
			<artifactId>poi</artifactId>
			<version>3.17</version>
		</dependency>
		<dependency>
			<groupId>org.apache.poi</groupId>
			<artifactId>poi-ooxml</artifactId>
			<version>3.17</version>
		</dependency>
		<dependency>
			<groupId>com.fasterxml.jackson.core</groupId>
			<artifactId>jackson-databind</artifactId>
			<version>2.12.2</version>
		</dependency>
		<dependency>
			<groupId>com.fasterxml.jackson.core</groupId>
			<artifactId>jackson-annotations</artifactId>
			<version>2.12.2</version>
		</dependency>
		<dependency>
			<groupId>org.apache.logging.log4j</groupId>
			<artifactId>log4j-api</artifactId>
			<version>2.17.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.logging.log4j</groupId>
			<artifactId>log4j-core</artifactId>
			<version>2.17.0</version>
		</dependency>
		<dependency>
			<groupId>com.aventstack</groupId>
			<artifactId>extentreports</artifactId>
			<version>4.1.5</version>
		</dependency>
		<dependency>
			<groupId>com.ohadr</groupId>
			<artifactId>ohadr.commons</artifactId>
			<version>0.3</version>
		</dependency>
		<dependency>
			<groupId>sax</groupId>
			<artifactId>sax</artifactId>
			<version>2.0.1</version>
		</dependency>
		<dependency>
			<groupId>org.apache.xmlbeans</groupId>
			<artifactId>xmlbeans</artifactId>
			<version>2.6.0</version>
		</dependency>
		<dependency>
			<groupId>org.assertj</groupId>
			<artifactId>assertj-core</artifactId>
			<version>1.4.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-jar-plugin</artifactId>
			<version>3.1.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven.plugin-tools</groupId>
			<artifactId>maven-plugin-annotations</artifactId>
			<version>3.5</version>
			<scope>provided</scope>
		</dependency>
		<!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple -->
		<dependency>
			<groupId>com.googlecode.json-simple</groupId>
			<artifactId>json-simple</artifactId>
			<version>1.1</version>
		</dependency>
		<dependency>
			<groupId>net.serenity-bdd</groupId>
			<artifactId>serenity-screenplay</artifactId>
			<version>${serenity.version}</version>
		</dependency>
		<dependency>
			<groupId>net.serenity-bdd</groupId>
			<artifactId>serenity-screenplay-webdriver</artifactId>
			<version>${serenity.version}</version>
		</dependency>
		<dependency>
			<groupId>org.json</groupId>
			<artifactId>json</artifactId>
			<version>20160810</version>
		</dependency>
		<dependency>
			<groupId>javax.xml.bind</groupId>
			<artifactId>jaxb-api</artifactId>
			<version>2.2.11</version>
		</dependency>
		<dependency>
			<groupId>org.glassfish</groupId>
			<artifactId>javax.json</artifactId>
			<version>1.0.4</version>
		</dependency>
		<!-- https://mvnrepository.com/artifact/jline/jline -->
		<dependency>
			<groupId>jline</groupId>
			<artifactId>jline</artifactId>
			<version>2.11</version>
		</dependency>
		<dependency>
			<groupId>commons-configuration</groupId>
			<artifactId>commons-configuration</artifactId>
			<version>1.6</version>
		</dependency>
		<dependency>
			<groupId>net.javacrumbs.json-unit</groupId>
			<artifactId>json-unit</artifactId>
			<version>2.10.0</version>
		</dependency>
		<dependency>
			<groupId>com.sun.xml.bind</groupId>
			<artifactId>jaxb-core</artifactId>
			<version>2.3.0</version>
		</dependency>
		<!-- https://mvnrepository.com/artifact/org.jfree/jfreechart -->
		<dependency>
			<groupId>org.jfree</groupId>
			<artifactId>jfreechart</artifactId>
			<version>1.5.0</version>
		</dependency>
		<dependency>
			<groupId>javax.activation</groupId>
			<artifactId>activation</artifactId>
			<version>1.1.1</version>
		</dependency>
		<dependency>
			<groupId>org.postgresql</groupId>
			<artifactId>postgresql</artifactId>
			<version>${postgresql.version}</version>
		</dependency>
		<dependency>
			<groupId>com.fasterxml.jackson.dataformat</groupId>
			<artifactId>jackson-dataformat-yaml</artifactId>
			<version>2.12.2</version>
		</dependency>
		<!-- https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl -->
		<dependency>
			<groupId>org.codehaus.jackson</groupId>
			<artifactId>jackson-mapper-asl</artifactId>
			<version>1.9.13</version>
		</dependency>
		<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
		<dependency>
			<groupId>com.fasterxml.jackson.core</groupId>
			<artifactId>jackson-core</artifactId>
			<version>2.12.2</version>
		</dependency>
	</dependencies>

Below is the reporting configuration from serenity.properties file

serenity.store.html.sourcer = true
serenity.logging = NORMAL
serenity.logging = QUIET
serenity.logging = VERBOSE
serenity.console.colors = true
serenity.requirement.types = feature,story
skip.ssl.validation = true
skip.email.notifications = false
startTime = 25.05.2022 08:27:39 PM
endTime = DONTCHANGEME
overall.status = DONTCHANGEME
newVar = false
report.file.location = 
report.contains = scenario
serenity.take.screenshots = FOR_FAILURES
serenity.reports.show.step.details = false

[wakaleo]

Try with version 3.2.5

[wee-jay]

Ok. Will try to accommodate new version and update here soon.

[wee-jay]

Tried with 3.2.5 version also and waited for around 2 hours.. but reports are not generated.

Our scripts stop here

[INFO] 
[ERROR] Tests run: 756, Failures: 158, Errors: 1, Skipped: 0
[INFO] 
[INFO] 
[INFO] >>> exec-maven-plugin:1.2.1:java (send-customized-mail) > validate @ pulse-qe-automation >>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:copy-resources (copy-resources) @ pulse-qe-automation ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] <<< exec-maven-plugin:1.2.1:java (send-customized-mail) < validate @ pulse-qe-automation <<<
[INFO] 
[INFO] 
[INFO] --- exec-maven-plugin:1.2.1:java (send-customized-mail) @ pulse-qe-automation ---
[INFO] Duration:::::0 Hours 45 Minutes 57 seconds 
log4j:WARN No appenders could be found for logger (org.apache.commons.configuration.ConfigurationUtils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

After this nothing happens and the scripts are like in stalled mode. We tried running on both local and on a remote host through pipeline.

[cliviu]

hi, a thread dump may be interesting at this point

[wee-jay]

When I ran with 35 test cases reports were generated

[ERROR] Tests run: 36, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] 
[INFO] >>> exec-maven-plugin:1.2.1:java (send-customized-mail) > validate @ pulse-qe-automation >>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:copy-resources (copy-resources) @ pulse-qe-automation ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] <<< exec-maven-plugin:1.2.1:java (send-customized-mail) < validate @ pulse-qe-automation <<<
[INFO] 
[INFO] 
[INFO] --- exec-maven-plugin:1.2.1:java (send-customized-mail) @ pulse-qe-automation ---
[INFO] Duration:::::0 Hours 2 Minutes 36 seconds 
log4j:WARN No appenders could be found for logger (org.apache.commons.configuration.ConfigurationUtils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[INFO] Sent message successfully....[INFO] 

[INFO] --- serenity-maven-plugin:3.2.5:aggregate (api-reports) @ pulse-qe-automation ---
[INFO] Test results for 36 tests generated in 5.1 secs in directory: 
[INFO] -----------------------------------------
[INFO]  SERENITY TESTS: ERROR
[INFO] -----------------------------------------
[INFO] | Test cases executed    | 15
[INFO] | Tests executed         | 36
[INFO] | Tests passed           | 35
[INFO] | Tests failed           | 0
[INFO] | Tests with errors      | 1
[INFO] | Tests compromised      | 0
[INFO] | Tests aborted          | 0
[INFO] | Tests pending          | 0
[INFO] | Tests ignored/skipped  | 0
[INFO] ------------------------ | --------------
[INFO] | Total Duration         | 3m 30s
[INFO] | Fastest test took      | 3s 950ms
[INFO] | Slowest test took      | 17s 538ms
[INFO] -----------------------------------------
[INFO] 
[INFO] SERENITY REPORTS

[wee-jay]

hi, a thread dump may be interesting at this point

Let me see

[wakaleo]

It could be memory or environment-related. We routinely generate reports for thousands of tests in a few minutes, so there is something else going on.

[wee-jay]

True.
We are also getting the reports generated but it is only when we tried reporting with higher amount of test cases.
Let me come back with the thread dump.

[wee-jay]

ThreadDump1.txt
Added Thread dump for your reference.

Let me know If heap dump is also needed will share on email as the file format of is not supported here.

[wee-jay]

OtherProcessThreadDump1.txt
OtherProcessThreadDump2.txt

There were two more unknown processes running.
So adding Thread dumps for those as well just in case if needed.

[wakaleo]

Thanks. I had a quick look, but I can't see any Serenity-related activity in any of these thread dumps.

[wee-jay]

Ok. Any suggestions..?

[wee-jay]

We were logging a lot of data during our script execution (around 900k lines) and I tried to disable these logs to see if they made this impact on the reports and after disabling the logs I got the report generated :)

[ERROR] Tests run: 756, Failures: 157, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] >>> exec-maven-plugin:1.2.1:java (send-customized-mail) > validate @ pulse-qe-automation >>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:copy-resources (copy-resources) @ pulse-qe-automation ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] <<< exec-maven-plugin:1.2.1:java (send-customized-mail) < validate @ pulse-qe-automation <<<
[INFO] 
[INFO] 
[INFO] --- exec-maven-plugin:1.2.1:java (send-customized-mail) @ pulse-qe-automation ---
[INFO] Duration:::::0 Hours 48 Minutes 49 seconds 
log4j:WARN No appenders could be found for logger (org.apache.commons.configuration.ConfigurationUtils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[INFO] Sent message successfully....
[INFO] 
[INFO] --- serenity-maven-plugin:3.2.5:aggregate (api-reports) @ pulse-qe-automation ---
[INFO] Test results for 721 tests generated in 42.8 secs in directory: 
[INFO] -----------------------------------------
[INFO]  SERENITY TESTS: FAILURE
[INFO] -----------------------------------------
[INFO] | Test cases executed    | 158
[INFO] | Tests executed         | 721
[INFO] | Tests passed           | 566
[INFO] | Tests failed           | 155
[INFO] | Tests with errors      | 0
[INFO] | Tests compromised      | 0
[INFO] | Tests aborted          | 0
[INFO] | Tests pending          | 0
[INFO] | Tests ignored/skipped  | 0
[INFO] ------------------------ | --------------
[INFO] | Total Duration         | 1h 14m 6s
[INFO] | Fastest test took      | 000ms
[INFO] | Slowest test took      | 21s 960ms
[INFO] -----------------------------------------
[INFO] 
[INFO] SERENITY REPORTS
[WARNING] No report found on classpath with name email
[INFO] 
[INFO] --- maven-resources-plugin:2.6:copy-resources (copy-favicon) @ pulse-qe-automation ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:copy-resources (copy-serenity-bdd-logo) @ pulse-qe-automation ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] 
[INFO] --- maven-cucumber-reporting:4.6.0:generate (web_mobile-reports) @ pulse-qe-automation ---
[INFO] About to generate Cucumber report.
Jun 04, 2022 12:05:20 PM net.masterthought.cucumber.ReportParser parseJsonFiles
[INFO] 
[INFO] --- maven-failsafe-plugin:3.0.0-M3:verify (api) @ pulse-qe-automation ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:35 h
[INFO] Finished at: 2022-06-04T12:05:32+05:30
[INFO] ------------------------------------------------------------------------

[wakaleo]

Interesting. 46 seconds to generate the reports sounds more normal. How were you generating the logs?

[wee-jay]

We were retrieving logs data using Restassured library .log().all() method for logs.

So I modified below statement from our code
response = executeService(method).log().all().extract().response();
To this
response = executeService(method).extract().response();
The executeService method is returning a ValidatableResponse type data which contained huge data and we were printing each line of headers, payloads, response, etc.

[wee-jay]

Still there is another culprit which is taking 20 to 30 min just to compile serenity reports and trigger email. It is an internal utility.. LOL .. I will work on it.
Thanks for your input and time.