Create dashboards for test results

[ycombinator]

Test results from CI runs on the master branch are being indexed into an Elastic-internal Elasticsearch cluster. It would be great to have some dashboards that visualize these results as a way of understanding progress over time + quickly determining packages that need more test coverage.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[sorantis]

+1 one of the integrations we're planning to add in the future is Jenkins (including dashboards). Do you see this as something related?

[ycombinator]

+1 one of the integrations we're planning to add in the future is Jenkins (including dashboards). Do you see this as something related?

I'm not sure. I imagine the Jenkins integration, at least to start off, might be fairly basic — in that it might involve ingesting metrics exposed by something like https://github.com/jenkinsci/metrics-plugin ? If so, I don't see anything specific to test results in there (not surprising as Jenkins is quite a generic workflow automation tool).

[mtojek]

@v1v Maybe you know where these metrics data can be stored? I tried the link posted in the observability robots repo, but it failed with:

{"statusCode":429,"error":"Too Many Requests","message":"[[parent] Data too large, data for [<http_request>] would be [2061694776/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2061694776/1.9gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, accounting=0/0b]: circuit_breaking_exception]: [parent] Data too large, data for [<http_request>] would be [2061694776/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2061694776/1.9gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, accounting=0/0b]"}

[v1v]

I'll create a dashboard similar to the ones we created for others and I'll raise a PR in the internal repository then you can review it

[mtojek]

SGTM!

EDIT:
It would be great to address requirements defined in this task: visualize total test coverage for integrations (pipeline tests, system tests, no tests). I understand if we need to collect more reference data.

[v1v]

It would be great to address requirements defined in this task: visualize total test coverage for integrations (pipeline tests, system tests, no tests). I understand if we need to collect more reference data.

Test coverage is not collected in the CI, but only, the build details, test stats summary and test failures.

For instance, given the build 561 then all the data is gathered and transformed in json files, and those files are the ones that are sent to ES, such as:

• ci-build-report.json , for the build details
• ci-test-report-bulk.json, for the test failures

ci-build-report.json aggregates some other files, such as: tests-summary.json and so on.

The index mapping can be found in -> https://github.com/elastic/apm-pipeline-library/blob/master/resources/ci-builds-mapping.json

[mtojek]

Build info is also useful (successful, skipped, failed tests + date) and would be great to setup a dashboard for it.

As the goal of this issue is to implement visualization for test coverage, do you have any recommendations in terms of Jenkins plugins? I can work on the missing transformation/scripting.

[v1v]

As the goal of this issue is to implement visualization for test coverage, do you have any recommendations in terms of Jenkins plugins? I can work on the missing transformation/scripting.

Test coverage can be visualise in Jenkins itself, for instance with https://plugins.jenkins.io/cobertura/

IIRC, some teams already use it:

• Specific for a build -> https://apm-ci.elastic.co/job/apm-server/job/apm-server-mbp/job/master/1087/cobertura/
• A trend -> https://apm-ci.elastic.co/job/apm-server/job/apm-server-mbp/job/master/cobertura/graph
• Another plugin -> https://apm-ci.elastic.co/job/apm-agent-dotnet/job/apm-agent-dotnet-mbp/job/master/560/coverage/

Regarding the transformation, there is a limitation with the existing implementation that runs as a post-build step, and it's related to how much data is ingested, that's the main reason we didn't send all the test data but the failed tests.