-
-
Notifications
You must be signed in to change notification settings - Fork 513
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[4.0.0-beta-2] Test result generated in 6+ minutes #3229
Comments
That's not normal - report generation should be in seconds (or minutes for test suites with > 10000 tests). There might be some kind of environment issue (such as an anti-virus) slowing down IO operations. Alternatively, see if you can do a thread dump or run the reporting task through a profiler you might be able to see where it is spending its time. These are more typical results that you would expect: If you can't figure it out we could have a look directly on your project as part of a commercial support package. |
@wakaleo Also I have the same issue. I am running a single Scenario and the report gets generated in 1 minutes. I am running the test in a quite huge project. We have in total of 116 feature files/440 Scenarios. But I am running a single Scenario. |
Report generation involves analysing the complete requirements hierachy, so even if you run a single scenario, it may take time due to the number of requirements. If it is a lot more than in previous versions, we would need to look at your project specifically to see why. |
Hi, I have the same issue with version 3.9.8. When going back to 3.8.1 the reports are generated as fast as before. I just ran with 3.9.8 and connected with jconsole to the JVM.
I honestly don't know, if this is completely normal or could mean something. I'm QA and not a programmer. |
Could you try with a more recent JDK? |
Switched to Java17. For me it's still slow with a lot of waiting and blocked threads:
|
In my case I'm using java 11, tests are running fast without the aggregate report, but when I do the |
Can you try with 4.0.0-beta-3? |
The 4.0.0-beta-3 is not available in the maven central repository. Can you publish the build there? |
It will be soon |
Works on my side, thanks @wakaleo |
I need to reopen this issued: |
Serenity reports taking more than 46 mins for two tests im using serenity with gradle plugin 3.9.8 Task :mdmtest:reports Task :mdmtest:aggregate Test Root: com.informatica.mdm |
serenity-maven-plug-in is released with Serenity-core, so should be the same version |
Try with 4.0.0-beta-3 when the gradle plugin is published. |
Any older stable version without this issue ? |
@wakaleo: Thanks for the fix. I upgraded to 4.0.0-beta-3 (all dependencies including plugin). |
I can confirm that starting from v. 3.9.8 to at least 4.0.0.-beta-2 the report generation takes very long time.
Can it be something similar that was already introduced in #2772? |
This should be fixed in 4.0.0-beta-3 |
It's till not too good - in 4.0.0-beta-3 reports were generated in 48 min compared to 4 hours in 4.0.0.-beta-2 and less than 4 min in 3.7.1. |
@Rangiferus Since I can't reproduce this it's very hard to troubleshoot. I really have no idea where to look: there may well be project or environment specific issues in play. If you are seeing this for a corporate project, the ideal would be to get one of the commercial support packages so we can investigate in detail on the project itself (see https://www.serenity-dojo.com/serenity-bdd-enterprise-support), otherwise I don't think I can do much unless I can see the issue reproduced. Alternatively, try running the reports with a profiler and see if you can see any particular areas that are causing delays. |
@wakaleo When i run tests with gradle failfast property as true , serenity report is generated with zero results ? Is there any tweaking need in build.gradle file when failfast is set to true ? |
No idea. An empty report happens when no Serenity json reports are found, so maybe Gradle fails before any Serenity tests get executed. |
In my case few tests are executed , i see json reports created as well. Only index.html shows zero tests |
What happens when you run gradle aggregate? |
5 tests completed, 1 failed, 1 skipped
FAILURE: Build failed with an exception.
BUILD FAILED in 9m 22s |
also i see this exception 5 tests completed, 1 failed, 1 skipped
|
@vinaypatil1994 It looks like the --fail-fast prevents Serenity from producing the Serenity JSON reports, and therefore makes it impossible to produce the HTML reports. I don't have a solution at this stage, but feel free to look at the Gradle plugin code to propose a PR, or request some commercial support - (see https://www.serenity-dojo.com/serenity-bdd-enterprise-support), if this feature would be valuable for your organisation. |
@wakaleo I think that the problem with the slow report generation is not project specific as I can run the same project with different Serenity versions and have totally different generation times.
The tests are executed the standard way
followed by the report generation with:
A report with one automated scenario is created in more than 7 minutes. |
@Rangiferus The dependencies look fine. But the reports should only take a few seconds to be generated. And since I'm not seeing this locally on Mac or Windows, or on the CI builds, I would need to investigate your project/environment more in detail to understand. That's what the commercial support options are for. |
I agree with @Rangiferus, this is not a project specific issue. I created a dummy project aprox 350 dummy tests (long live github copilot). using serenity 3.2.4 -> takes ~ 16 seconds Being a beta version I will rollback to 3.2.4 and wait for new releases. |
Project specific or not, if it is valuable to your organisation, and you would get value from seeing this delivered more quickly, since this is an open source project you can either investigate and contribute a PR, or get commercial support. Open source does not mean free, and most major features are sponsored, not written out of charity. |
Works using 4.0.0-beta-4 🥇 |
Generating report takes a lot of time in my case 8 minutes, and I run only 2 test using specific tags.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO] GENERATING SUMMARY REPORTS...
[INFO] GENERATING REQUIREMENTS REPORTS...
[INFO] GENERATING RESULT REPORTS...
[INFO] GENERATING ERROR REPORTS...
[INFO] Test results for 2 tests generated in 6 mins and 24.9 secs in directory: file:/..../target/site/serenity/
[INFO] ------------------------------------------------
[INFO] | SERENITY TESTS: | SUCCESS
[INFO] ------------------------------------------------
[INFO] | Test scenarios executed | 2
[INFO] | Total Test cases executed | 2
[INFO] | Tests passed | 2
[INFO] | Tests failed | 0
[INFO] | Tests with errors | 0
[INFO] | Tests compromised | 0
[INFO] | Tests aborted | 0
[INFO] | Tests pending | 0
[INFO] | Tests ignored/skipped | 0
[INFO] ------------------------------- | --------------
[INFO] | Total Duration| 8s 151ms
[INFO] | Fastest test took| 3s 610ms
[INFO] | Slowest test took| 4s 541ms
[INFO] ------------------------------------------------
[INFO]
[INFO] SERENITY REPORTS....../target/site/serenity/index.html
[INFO]
[INFO] --- failsafe:3.0.0-M5:verify (default) @ ....... ---
[WARNING] Parameter 'encoding' (user property 'encoding') is deprecated: since of 2.20.1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:43 min
[INFO] Finished at: 2023-08-17T11:44:50+03:00
This is my POM:
`
4.0.0
The text was updated successfully, but these errors were encountered: