Enable test coverage calculation for integrations #1344

mtojek · 2021-07-22T12:31:23Z

Fixes: #755

This PR updates Jenkinsfile to enable test coverage calculation for integrations (all test types).

Coverage report for AWS:

Coverage report for packages:

elasticmachine · 2021-07-22T13:41:12Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-07-26T15:33:27.889+0000
Duration: 70 min 10 sec
Commit: 161b033

Test stats 🧪

Test	Results
Failed	0
Passed	2600
Skipped	3
Total	2603

Trends 🧪

mtojek · 2021-07-22T14:06:21Z

/test

mtojek · 2021-07-22T14:11:33Z

Some weird exceptions appeared. I'm verifying if they are reproducible:

[2021-07-22T13:55:26.157Z] [Cobertura] Publishing Cobertura coverage report...
[2021-07-22T13:55:26.157Z] 
[2021-07-22T13:55:26.415Z] [Cobertura] Publishing Cobertura coverage results...
[2021-07-22T13:55:26.415Z] 
[2021-07-22T13:55:26.444Z] FATAL: Unable to parse /var/lib/jenkins/jobs/Ingest-manager/jobs/integrations/branches/PR-1344/builds/2/coverage1.xml
[2021-07-22T13:55:26.444Z] hudson.util.IOException2: Cannot parse coverage results
[2021-07-22T13:55:26.444Z] 	at hudson.plugins.cobertura.CoberturaCoverageParser.parse(CoberturaCoverageParser.java:86)
[2021-07-22T13:55:26.444Z] 	at hudson.plugins.cobertura.CoberturaCoverageParser.parse(CoberturaCoverageParser.java:52)
[2021-07-22T13:55:26.444Z] 	at hudson.plugins.cobertura.CoberturaPublisher.perform(CoberturaPublisher.java:594)
[2021-07-22T13:55:26.444Z] 	at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123)
[2021-07-22T13:55:26.445Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:99)
[2021-07-22T13:55:26.445Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:69)
[2021-07-22T13:55:26.445Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2021-07-22T13:55:26.445Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2021-07-22T13:55:26.445Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-07-22T13:55:26.445Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2021-07-22T13:55:26.445Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2021-07-22T13:55:26.445Z] 	at java.lang.Thread.run(Thread.java:748)
[2021-07-22T13:55:26.445Z] Caused by: org.xml.sax.SAXParseException; lineNumber: 21; columnNumber: 12; Content is not allowed in trailing section.
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1473)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$TrailingMiscDriver.next(XMLDocumentScannerImpl.java:1431)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
[2021-07-22T13:55:26.445Z] 	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:327)
[2021-07-22T13:55:26.446Z] 	at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
[2021-07-22T13:55:26.446Z] 	at hudson.plugins.cobertura.CoberturaCoverageParser.parse(CoberturaCoverageParser.java:78)
[2021-07-22T13:55:26.446Z] 	... 11 more
[Cobertura] Unable to parse /var/lib/jenkins/jobs/Ingest-manager/jobs/integrations/branches/PR-1344/builds/2/coverage1.xml

mtojek · 2021-07-22T14:12:52Z

Another issue - as we run isolate tests of every package on separate hosts, results aren't concatenated properly:

mtojek · 2021-07-22T14:19:53Z

I'm not sure if the exception isn't related to overriding coverage reports (will try to solve this one first). @v1v Do you know if there is any argument for coverageReport() to properly this use case or should I use any temporary bucket to store coverage results.

EDIT:

It looks like we need to involve stashV2 and unstashV2.

v1v · 2021-07-22T14:52:10Z

It looks like we need to involve stashV2 and unstashV2.

Exactly stashV2 is an adhoc temporary stash implementation with google storage, while stash is the built-in that stores in the jenkins controller, this latter doesn't scale well I'm afraid with parallel stages.

If you can use that approach most likely it should work :)

mtojek · 2021-07-22T14:59:26Z

Thank you for quick response, Victor!

Actually I'm considering to use googleStorageUpload and googleStorageDownload directly, so that I can store all files in the same bucket ("drop point" for coverage results, every parallel job uploads coverage results with googleStorageUpload) and then in post/cleanup just call googleStorageDownload with an asterisk (TBC?), so that with one call to Google Storage I can download all coverage files and publish them with coverageReport(). Please let me know what do you think.

I'm happy to switch to magic from apm-pipeline-library if there is a better implementation hidden :)

v1v · 2021-07-22T15:53:38Z

Actually I'm considering to use googleStorageUpload

Don't use googleStorageUpload if the pipeline runs in parallel multiple stages, there is a known bug with that plugin. You can use the googleStorageUploadExt step instead. See https://github.com/elastic/apm-pipeline-library/blob/master/vars/googleStorageUploadExt.txt

googleStorageDownload works fine.

This reverts commit 7db603f.

This reverts commit 88dd12e.

mtojek · 2021-07-26T15:42:14Z

I think we can review calculation rules and adjust weights if necessary. I understand it's a bit misleading that it assumes "lack of tests" as 0 and N tests as N.

Currently it's possible to have:

package with 4 tests (1 pipeline, 1 system, 1 asset, 1 static) = 100% coverage
another package only with 97 pipeline tests = 97% coverage.

EDIT:

Maybe we should consider only binary rates (0 vs 1) for presence of any tests? Package examples above would be considered as:

4 (all) different test types used = 100% coverage
only pipeline tests present = 25% coverage

Asset tests for package (not data streams), marked as "-" would be skipped in the calculation.

WDYT?

v1v

💯

jsoriano · 2021-07-27T12:25:57Z

Regarding calculations, I think I like more the second option, but thinking on the files covered by the tests instead of based on the presence of tests, so a data stream without pipelines can have 100% coverage without pipeline tests.

Update dependency on elastic-package

83e4dad

mtojek self-assigned this Jul 22, 2021

Bugfix

3a77874

mtojek marked this pull request as draft July 22, 2021 14:31

Merge branch 'master' into update-ep-0722

3fba3ad

mtojek changed the title ~~Update dependency on elastic-package~~ Enable test coverage calculation for integrations Jul 26, 2021

mtojek added 16 commits July 26, 2021 09:20

Stubs for stash/unstash

46762a7

For testing only

7db603f

Use Google Storage to save coverage files

bc45cb8

JOB_GCS_EXT_CREDENTIALS

218f21c

Fix

9bf3ac4

bucketUri

dbd9edb

Fix

5ac56ab

Fix

727dd14

More path fixes

5e56878

WIP

a43e77f

Missing credentials

24a9448

Try: test not executed

88dd12e

Try: publish coverage reports

3c18304

Try

7bcce8a

Refactoring

469f2c5

isCoverageReportPresent

ee84304

mtojek added 5 commits July 26, 2021 14:50

Comment

8e7b735

Revert "For testing only"

63d1f95

This reverts commit 7db603f.

Revert "Try: test not executed"

9bc01bf

This reverts commit 88dd12e.

Add missing coverageReport call

25ff4d6

Cleanup

161b033

mtojek requested review from jsoriano and v1v July 26, 2021 15:33

mtojek marked this pull request as ready for review July 26, 2021 15:36

v1v approved these changes Jul 27, 2021

View reviewed changes

jsoriano approved these changes Jul 27, 2021

View reviewed changes

mtojek merged commit 72696fb into elastic:master Jul 27, 2021

Enable test coverage calculation for integrations #1344

Enable test coverage calculation for integrations #1344

Uh oh!

Conversation

mtojek commented Jul 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Jul 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

Uh oh!

mtojek commented Jul 22, 2021

Uh oh!

mtojek commented Jul 22, 2021

Uh oh!

mtojek commented Jul 22, 2021

Uh oh!

mtojek commented Jul 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

v1v commented Jul 22, 2021

Uh oh!

mtojek commented Jul 22, 2021

Uh oh!

v1v commented Jul 22, 2021

Uh oh!

mtojek commented Jul 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

jsoriano commented Jul 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mtojek commented Jul 22, 2021 •

edited

Loading

elasticmachine commented Jul 22, 2021 •

edited

Loading

mtojek commented Jul 22, 2021 •

edited

Loading

mtojek commented Jul 26, 2021 •

edited

Loading