-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: Setup codecov to block PRs from getting merged if they decrease coverage #139
Conversation
Skipping CI for Draft Pull Request. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just notice this is still in draft.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Let's try and see how it goes.
https://issues.redhat.com/browse/STONEBLD-632
Implemented gating pull requests if they decrease overall test coverage by more than 1 %.
This threshold is subject to change based on feedback, but should cover most false negatives.
Behavior when removing covered lines is subject to change too. see possible alternatives
codecov/project
check should be made mandatory by repo admins if lowering test coverage should actually block the ability to merge the PR.Now ignoring
pkg/github
andpkg/gitlab
because measuring their test coverage is not desired.Also split Codecov upload into its own job, so it can be re-ran without re-running the tests. This job should fail in case of an error during upload.
Please see some examples of PRs:
decreased coverage
increased coverage, but overall decrease due to unexpected changes, still passed
See how test coverage can differ even between the same exact code:
coverage decreased even though it should not have changed, still passed
please compare with exact same code in a different PR, but with a different test coverage