retain html report

[gravityvi]

Changes

• Add a flag to retain HTML report across test runs
• Default behaviour is not changed it generates different HTML reports if retain_report is set to true

Impacts

• fixes Retain HTML report across test runs #3307

[vaibhavsingh97]

Do we really need a flag? What would be the best scenario is that we append the timestamp in the index.html file which will be unique, as well as let us know which file has the latest report

[gravityvi]

We can't change index.html as it would be used in different CI pipelines to host the HTML page. In future we would need to maintain a single index.html containing all the results from previous test runs. We need a flag to preserve the default behavior and not create breaking changes for those using HTML reporter.

[beatfactor]

I think we don't need a flag. The user can just define a folder_format function that will be undefined by default, in which case the current behaviour stays.

We should probably use reporter as the settings name, because we might have other options there. We can also in the future move the output_folder setting in it and have something like:

reporter: {
  output_folder: '',
  folder_format: function() {}
}

[beatfactor]

or it can be a function that returns the default value, as you've implemented here.

[beatfactor]

I don't think we should have this as prefix, as it will make it harder to sort the report folder names.

[gravityvi]

default output_folder would is now moved inside reporter_options however output_folder provided directly under config would also be considered.

If folder_format is provided Nightwatch would retain HTML reports across test runs. For example

// nightwatch.conf.js
{ 
 reporter_options: {
    output_folder: 'test_outputs',
    folder_format: () => Date.now()
 }
}