-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feat][ci] Collect code coverage for integration tests from docker containers #19263
[feat][ci] Collect code coverage for integration tests from docker containers #19263
Conversation
@yaalsn Please review this PR. |
d69da8f
to
d0948cc
Compare
d0948cc
to
23c1865
Compare
Example of code coverage results for an earlier experiment: https://app.codecov.io/github/lhotari/pulsar/commit/369dec198b5cf6b98cd5c52a0c32126974a8e9ac/tree |
Codecov results of the currently running PR build in the fork will be available at https://app.codecov.io/github/lhotari/pulsar/commit/23c1865976414a166e4ff2d6c5874937390cfc9e/tree |
#19264 will also be needed to fix coverage results. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is awesome!
/pulsarbot rerun-failure-checks |
I wonder if including integration tests in the code coverage report would hide gaps in unit test coverage (unless the latter coverage metric was already broken :/) Is there a possibility to separate the coverage report for unit vs integ tests? |
Yes, it should be possible to view unit test coverage separately. They are reported to Codecov with different flags. |
@aymkhalil The docs for the flags feature in Codecov: https://docs.codecov.com/docs/flags . It seems that the full feature is disabled by default. |
To experiment, I enabled it for my fork at https://app.codecov.io/github/lhotari/pulsar/flags . For apache/pulsar , I don't have access so I'll have to create a ASF ticket for handling that. |
Motivation
Code coverage metrics don't currently consider integration tests and systems tests that are run as part of the Pulsar CI pipeline. There hasn't been a way to measure the total code coverage of Pulsar CI pipeline tests although it would be useful to see what parts of Pulsar code base are tested and what aren't.
Modifications
build/run_integration_group.sh
script.Documentation
doc
doc-required
doc-not-needed
doc-complete
Matching PR in forked repository
PR in forked repository: lhotari#134