Ability to merge ndjsons generated by formatters into an html report

[Santonat316]

When having multiple stages of running UI/API tests, the ability to combine the ndjsons produced in different stages into a single html report is quite useful. Is there a plan to implement this feature in future Reqnroll releases?

[hideintheclouds]

I think that in order to have feature parity with SpecFlow.Plus.LivingDoc.CLI, the capability of merging multiple ndjson files into a single HTML report is needed.
Since the Reqnroll tests are executed in Pipelines, it will be beneficial to have only 1 HTML file instead of multiple HTMLs corresponding to each run.

Additionally, this will be needed in air-gapped environments, since tools such as cucumber html-formatter will produce HTML files and try to fetch resources (css, js) from Internet.
The Reqnroll HTML report seems to be self-contained, so it is the perfect fit in such environments.

[gasparnagy]

We have seen the problem, but no concrete plans yet.

Do you have some ideas how this should work from the user's perspective?

(Everyone, please use the vote button to indicate your interest to proposed feature ideas, like this.)

[304NotModified]

Isn't merging ndjson files super easy? It's just concatenation?
Or do I misunderstood the issue?

E.g.

Powershell:

Get-Content file1.ndjson, file2.ndjson | Set-Content merged.ndjson

Bash

cat file1.ndjson file2.ndjson > merged.ndjson

[gasparnagy]

Does the HTML report shows the details of before after test run hooks at all?

Another interesting test situation is when

• the two runs contain different scenarios of the same feature
• the two runs contain results of the same scenario (like for vs test rerun)

[clrudolphi]

global_hooks.html
The attached html is the result of joining the results of two executions of the same test. I manually copied the messages from one html file to another. The collection of ndjson messages embedded in the html includes two TestRunStarted/Finished pairs, two Meta messages, etc.

This exercise used the oob html rendering; I did not attempt to modify the template.

A few observations that result from this simple approach:

• Global hooks (Before/After TestRun and/or Before/After Feature) are included in the rendering, but at the bottom and without context (which test run they were from) - they appear to be rendered in the order they were seen in the file.
• Scenarios are grouped by Feature URI, which we populate with the feature code-behind filename (ie, it includes the feature name and a GUID). Therefore, the same scenarios executed in different runs are not grouped together.
• The test HTML header appears to be populated on the last Meta and TestRun seen in the file. I note this as this test has one set of messages from a test that was executed two weeks ago, and one that was executed today. The header is based off of information from today's run.

While not tested in this exercise, if we did have different scenarios of the same feature, they would have been kept separate as they would have belonged to two different feature URIs.

[clrudolphi]

• Scenarios are grouped by Feature URI, which we populate with the feature code-behind filename (ie, it includes the feature name and a GUID). Therefore, the same scenarios executed in different runs are not grouped together.

@gasparnagy Any interest in changing our current behavior on how the contents of the FeatureURI is populated when creating ndjson? Should this instead just be the Feature name rather than the feature file name?

[gasparnagy]

@gasparnagy Any interest in changing our current behavior on how the contents of the FeatureURI is populated when creating ndjson? Should this instead just be the Feature name rather than the feature file name?

I think I don't understand this fully yet to be able to answer. Could you please link the code that generates this URI currently? Why don't we see the GUID like what you have in the linked HTML report (FeatureFile2404e07aab164db48da67a07291a8590.feature) when we do a normal execution?

[clrudolphi]

never mind. My own fault.
I was using as samples the ndjson and html files generated by the Formatters tests. These use the TestGenerator to add the feature to the generated test project and it is adding the GUIDs to the names.

This is not an issue for regular users. Sorry for raising a confusing false issue.

[clrudolphi]

Here is a sample project on how to use the Cucumber.Messages and Cucumber.HtmlFormatter projects to create a CLI program that can generate html files from ndjson files and support merging of ndjson files into a single html report.