Add HTML reporter

[marevers]

There is an official HTML reporter for Newman that creates reports in HTML (https://github.com/postmanlabs/newman-reporter-html).

This PR adds newman-reporter-html as a dependency and serves up the generated HTML report(s) at /htmlreport/<probe>. Links to the HTML report are also shown on the root page (only if they are enabled in the probeConfiguration).

To enable HTML report for a collection, options.reporter.html.export and options.reporters: 'html' must be set.

Example:

module.exports = {
    "collection": {
        options: {
            collection: require('./collection.json'),
            environment: require('./environment.json'),
            timeoutRequest: 30000,
            reporters: 'html',
            reporter: {
              html: {
                export: './collectionHtmlReport.html'
              }
            }
        }
    }
};

[hudeldudel]

Hi Martijn,
thank you for contributing!

But I have some concerns with this pull request.

1. Dependencies
   The latest version of newman-reporter-html depends on newman version 4 but postman-exporter is using version 5.

npm ERR! Could not resolve dependency:
npm ERR! peer newman@"4" from newman-reporter-html@1.0.5

See also postmanlabs/newman-reporter-html#255

I could force newman-reporter-html to use newman 5, but I really don't want to.

It also looks very stale. Last release is 3 years old.

May be, newman-reporter-htmlextra is a better alternative to use? It seems to be better maintained but I did not have a closer look to it.

2. Concurrency

/htmlreport/:probe internally writes to a file with a static name. Concurrent requests may return the wrong response.

3. Strategic focus
   The main intention of postman exporter is to provide metrics for Prometheus. Also generating HTML reports adds dependencies for an additional, non-intended function.
   But since I find this additional function quite nice, I would still include it if the dependency issues mentioned above can be solved and do not include too much stuff. :-)

[marevers]

Hi Sebastian,

thank you for replying to my PR. Please find answers below:

1. Dependencies
   The latest version of newman-reporter-html depends on newman version 4 but postman-exporter is using version 5.

npm ERR! Could not resolve dependency:
npm ERR! peer newman@"4" from newman-reporter-html@1.0.5

See also postmanlabs/newman-reporter-html#255

I could force newman-reporter-html to use newman 5, but I really don't want to.

It also looks very stale. Last release is 3 years old.

May be, newman-reporter-htmlextra is a better alternative to use? It seems to be better maintained but I did not have a closer look to it.

Thanks for the hint, I will take a look at that. I failed to notice the dependency is that old, I will take a look at the one you mentioned. It did work for me for some reason, but that might have been because I tested using the Dockerfile only.

2. Concurrency

/htmlreport/:probe internally writes to a file with a static name. Concurrent requests may return the wrong response.

If I understand correctly, that file is written to (overwritten) every time the metrics for that probe are generated and as such will always reflect the last known state from the last scrape from Prometheus. I thought this would be useful, to have that in sync for debugging / reporting purposes. The call to /htmlreport/:probe does not actually generate the report so a concurrency issue should not be a thing, unless of course one would incorrectly configure it and have two probes/collections use the same filename.

3. Strategic focus
   The main intention of postman exporter is to provide metrics for Prometheus. Also generating HTML reports adds dependencies for an additional, non-intended function.
   But since I find this additional function quite nice, I would still include it if the dependency issues mentioned above can be solved and do not include too much stuff. :-)

Thank you. I thought of it as an auxiliary function to the Prometheus metrics, therefore also my answer to 2. I will have a look at alternatives for the reporter dependency.

[marevers]

I updated package-lock.json with the dependencies for newman-reporter-htmlextra.

[hudeldudel]

Sorry for the delay. I've looked at it again and have a few comments and questions.

From my point of view it doesn't work as expected.

My expectation would be that /htmlreport/:probe would run a collection and return the associated HTML report instead of the Prometheus metrics. In fact, no collection is executed at all, but the HTML report of the last execution of /probe/ is returned.

/htmlreport/:probe doesn't even has a relation to :probe.
Sample flow:
/probe/example -> runs example, returns Prometheus metrics and writes the HTML report as a file.
/probe/another-example -> runs another-example, returns Prometheus metrics and overwrites the file with the report for another-example
/htmlexport/example -> returns the HTML report for another-example.

What is the behavior you actually want?
a) The ability to call an endpoint that runs a collection and returns the result as an HTML report.
b) The ability to access the result of a previously executed collection again afterwards and choose whether to output Prometheus metrics or an HTML report for easier debugging.

b) would be somewhat analogous to the history in Prometheus Blackbox Exporter. Postman Exporter currently doesn't even has a history.

a) would have to be implemented differently. Instead of a fixed file name, a temporary id should be used for each call that generates a report, returns it, and then deletes the file again.
Of course, it would be even better not to write the report to a file at all, but directly to the response of the web server. The way I see it, however, newman-reporter-htmlextra cannot do this, since it is called by Newman and there is no return option via this path.

[marevers]

Thank you for your reply and guidance. Please find my comments/answers below:

From my understanding there is a relation between /htmlreport/:probe and :probe, owing to the dedicated link that needs to be set per collection. As an example:

module.exports = {
    "collection1": {
        options: {
            collection: require('./collection1.json'),
            environment: require('./environment1.json'),
            timeoutRequest: 30000,
            reporters: 'htmlextra',
            reporter: {
              htmlextra: {
                export: './collection1HtmlReport.html'
              }
            }
        }
    },
    "collection2": {
        options: {
            collection: require('./collection2.json'),
            environment: require('./environment2.json'),
            timeoutRequest: 30000,
            reporters: 'htmlextra',
            reporter: {
              htmlextra: {
                export: './collection2HtmlReport.html'
              }
            }
        }
    }
};

As such, there are two collections, each using their own identifiers (collection1 & collection 2), collection files, environment files and paths for the html report. In that way /htmlreport/collection1 will return the last results for collection1 and /htmlreport/collection2 will return the last results for collection2. There is potential for misconfiguration, though, as setting reporter.htmlextra.export to the same filename will result in the behavior you described: whatever collection was executed last, will be the one returned as both are written to the same file.

One optimization there could be to take away that control from the user and instead generate a filename that is based on the collection name without allowing the user to configure it. If you think that could fix the issue, I can take a look at that.

What is the behavior you actually want?
a) The ability to call an endpoint that runs a collection and returns the result as an HTML report.
b) The ability to access the result of a previously executed collection again afterwards and choose whether to output Prometheus metrics or an HTML report for easier debugging.

I tend more towards option b, for two reasons:

1. Some longer collections can run for quite a while, in our experience sometimes up to 30 seconds. If the functionality is implemented in a synchronous way, when calling /htmlreport/:probe causes the collection to run, the user would need to wait until the collection is finished. And as every call runs the collection again, the wait is always necessary. With option b, as metric scraping (and HTML report generation) happens in the background, the user is presented the (last) results immediately.
2. Allowing the collection to run on demand every time /htmlreport/:probe is called gives a degree control to a user that the user maybe should not have. From a usecase perspective, we would allow a greater subset of users to look at the reports, while only Prometheus itself can use /probe/:probe and thus, execute the collections. Some collections should maybe not be ran too often / in too quick succession - and allowing the user to control this would mean some other means of rate limiting could be necessary.

In my opinion this would not need to be extended to outputting Prometheus metrics for a previously executed collection, as these would be available in Prometheus anyway, having been scraped before.

I would love to hear your thoughts on this.