Try rounding codecov down instead of up - #1413
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1413 +/- ##
=======================================
Coverage 100.0% 100.0%
=======================================
Files 213 213
Lines 13936 13936
=======================================
Hits 13929 13929
Misses 7 7 Continue to review full report at Codecov.
|
| @@ -1,3 +1,3 @@ | |||
| coverage: | |||
| round: up | |||
| round: down | |||
There was a problem hiding this comment.
Maybe we need to change the precision? I think rounding down is the default behavior and we were seeing issues like what is happening in #1388 before we changed to rounding up.
There was a problem hiding this comment.
@freddyaboulton ah you are correct, this is the default behavior...
There was a problem hiding this comment.
The codecov.io docs are super unclear about what "round" means:
Which direction to you want to round the coverage value
They should have included an example... what's their algorithm for determining whether or not a job passes codecov? I think the algorithm is
- Compute coverage percentage before and after
- Take the signed difference, after - before
- Round that difference using the
roundpolicy, then truncate usingprecisiondigits after the decimal - If the resulting number is less than 0, fail
But... I haven't found their docs which say whether or not this is true, hence further trial and error
There was a problem hiding this comment.
Oops I spoke too soon:
round((hits / (hits + partials + misses)) * 100, 5) = Coverage Ratio
According to this, rounding up means a higher coverage ratio, which I think is what we want. So I'll decrease the precision but leave this set to "up". Does that make sense?
d5c2bc9 to
43d5365
Compare
| round: up | ||
| precision: 2 | ||
| round: down | ||
| precision: 1 |
… we rely on linux unit tests to report coverage
|
|
||
| codecov: | ||
| notify: | ||
| wait_for_ci: true |
There was a problem hiding this comment.
This will theoretically only run a codecov report once CI is green. We'll see :)
There was a problem hiding this comment.
No more premature +/ 8% changes in coverage? 😃
|
@jeremyliweishih I think that is also worth looking into! I don't think rounding can save us if we ever decide to delete a whole file or something like that. |
| status: | ||
| project: | ||
| default: | ||
| threshold: 1% |
There was a problem hiding this comment.
@jeremyliweishih good point! Let's try it out. I haven't been able to find docs for threshold other than the link you found.
I saw someone on their forum used this and didn't complain about it breaking, beyond some sort of caching bug which is hopefully now irrelevant.
dsherry
added
testing
|
I just tested this out in a PR #1414 , deleted a unit test to trigger a <0.01% coverage drop, and the codecov job was marked as passing! We may wanna decrease the threshold to 0.1% or something, but this is fine for now. Merging soon! |
4 participants
Perhaps I had this backwards previously? I hope this helps with the recent codecov flakes we've been seeing occur when people delete lines of code.
https://docs.codecov.io/docs/codecovyml-reference#coverageround