-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Carryover codecov coverage #680
Conversation
Flagging is basically a way to give a category to unittests (grouping them together)
Given that we have a monorepo, we do not run all tests every time. carryover makes codecov use any previous run as an oracle if a submission is missing coverage data for an existing flag
Codecov Report
@@ Coverage Diff @@
## main #680 +/- ##
===========================================
- Coverage 58.67% 32.05% -26.62%
===========================================
Files 103 302 +199
Lines 14430 35156 +20726
Branches 2374 6141 +3767
===========================================
+ Hits 8467 11270 +2803
- Misses 5632 23362 +17730
- Partials 331 524 +193
Flags with carried forward coverage won't be shown. Click here to find out more. |
The drop in coverage of this PR is "fake", last run on main only executed checkbox-ng tests, and due to the missing carryover, now codecov thinks that we have a 5x% coverage on main |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that for providers the "provider-base" would be much better than just "base".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome, +1
Description
Given that we have a monorepo, we have several "parts" that are independent in the same spot. Every time a change is pushed to a pr or main, if a part was changed, it is tested. Once a part is tested the coverage results are pushed to codecov.io.
This fixes this issue by enabling carryover and flagging the parts. Carryover basically means that if a flag (part) is not updated in a submission, the previous value is used instead. Flags are how codecov identifies the parts, this creates one per part + one for all
unittests
Resolved issues
The issue that we are having is that if a part is not tested in a submission (for example, because it was not changed), codecov "forgets" it existed right now, leading to the coverage values that are not reflecting of reality.
Documentation
This adds a new flag per part, this should make the output more readable without any need for additional documentation.
Tests
See the codecov bot comment to get to the dashboard, it should now have a "flags" section that lists the coverage of each part