Code coverage randomly drops when it includes folders that had no tests before #672
Assignees
Labels
area/testing
Related to any test or testing suite
ci/build
This is issue is about the build or CI system, and the administration of it.
code quality
Related to improving code quality
Milestone
Comments
shahzadlone
added
area/testing
4 tasks
shahzadlone
added a commit
that referenced
this issue
Jul 28, 2022
Resolves #672 **Description:** Ensures the empty packages aren't left out (because previously packages with 0 tests won't be included in the coverage). This should help avoid seeing the random coverage drops we see in PRs that are actually introducing tests. __Moreover__ - Removes the make rule `make test:coverage-quick`. - Change make rule `make test:coverage-full` to `make test:coverage` - Adds the ability to also provide `path=...` to `make test:coverage` like we had for `make test:coverage-html`. - Account for integration test coverage for the rule `make test:coverage-html`. __Limitation__ Using `-coverpkg` does not actually help if the package is never called in any test at all. `-coverpkg` is a cover-up that allows packages from a different package to provide coverage for this package. So, say package `a` has no tests, and thus no coverage. But package `b` has tests, and those tests call into `a`. As a result, if you start using a `-coverpkg` that covers both packages, the tests from package `b` will report its coverage of package `a` as coverage for `a`. However, if we have a package `c` that is never called by either `a` or `b` then that package will still be left out in the dry, even if it would match the `-coverpkg` given. Reference: golang/go#24570
shahzadlone
added a commit
to shahzadlone/defradb
that referenced
this issue
Feb 23, 2024
Resolves sourcenetwork#672 **Description:** Ensures the empty packages aren't left out (because previously packages with 0 tests won't be included in the coverage). This should help avoid seeing the random coverage drops we see in PRs that are actually introducing tests. __Moreover__ - Removes the make rule `make test:coverage-quick`. - Change make rule `make test:coverage-full` to `make test:coverage` - Adds the ability to also provide `path=...` to `make test:coverage` like we had for `make test:coverage-html`. - Account for integration test coverage for the rule `make test:coverage-html`. __Limitation__ Using `-coverpkg` does not actually help if the package is never called in any test at all. `-coverpkg` is a cover-up that allows packages from a different package to provide coverage for this package. So, say package `a` has no tests, and thus no coverage. But package `b` has tests, and those tests call into `a`. As a result, if you start using a `-coverpkg` that covers both packages, the tests from package `b` will report its coverage of package `a` as coverage for `a`. However, if we have a package `c` that is never called by either `a` or `b` then that package will still be left out in the dry, even if it would match the `-coverpkg` given. Reference: golang/go#24570
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We have now noticed a few times where the code coverage drops unexpectedly due to a certain package not being tracked previously due to it not having any tests, now when you introduce new tests to a package, the coverage starts taking that entire package into account resulting in a drop in test coverage despite having added more tests.
Solution, find a way to always include all packages into code coverage (minus any we explicitly tell code cov to ignore)even if they have no tests.
Perhaps by adding
-coverpkgflag.The text was updated successfully, but these errors were encountered: