[testing] investigate coverage inconsistency. #14756
Assignees
Labels
area/testing
kind/engineering
Work that is not visible to an external user
resolution/fixed
This issue was fixed
Milestone
Comments
dixler
added
area/testing
kind/engineering
dixler
changed the title
This was referenced Dec 6, 2023
github-merge-queue bot
pushed a commit
that referenced
this issue
Dec 8, 2023
Fixes #14756 Flaky tests cause coverage to be rerun and drops coverage data making CodeCov data less reliable/usable. One of the most commonly flaky tests is `TestCreatingProjectWithEmptyConfig` in `pkg` which contains a large number of tests. Moving this test to the top of the file appears to lead to less flakiness.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
Pulumi CodeCov reports still fluctuate across runs.
source_query.go is reported with 0% coverage despite the following tests https://github.com/pulumi/pulumi/blob/0b250fcf2a881aa1e73582e6e477749d31af9062/pkg/resource/deploy/source_query_test.go
I suspect gotestyourself/gotestsum#274 is the culprit as gotestsum re-runs failed tests and prior passed tests are ignored.
Example
https://github.com/pulumi/pulumi/actions/runs/7100733481/job/19327776558
CI / Unit Test / pkg 2/7 on ubuntu-latest/current
DONE 2 runs, 852 tests, 9 skipped, 9 failures in 670.888s
CodeCov report (0% source_query.go)
https://app.codecov.io/github/pulumi/pulumi/commit/0b250fcf2a881aa1e73582e6e477749d31af9062/blob/pkg/resource/deploy/source_query.go
Output of
pulumi aboutn/a
Additional context
No response
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
The text was updated successfully, but these errors were encountered: