fix test coverage reporting when running with parallelization #6590
Conversation
There was a problem hiding this comment.
Al, this looks really good. Amazing bit of sleuthing!
Just a couple things to make the change history easier to follow for future developers: 1. Can you fix up the linting fixes to previous commit and 2. the PR description is great, could you add that to the commit message. Thanks!
aldavidson
force-pushed
the
fix-parallel-test-coverage
branch
from
4a0c7d2
to
1122eb7
Compare
Done! |
There was a problem hiding this comment.
Great work @aldavidson ⭐
It's odd that your test run in the PR description seemed to produce 93.86% coverage, whereas Jenkins claims it to be 94.55%. I wonder if it's the difference between running MiniTest & Cucumber vs just Cucumber on its own, since Jenkins does both but I think you only ran Cucumber.
Either way, looks great. Thanks for this!
That may also be because I rebased from main after writing the description? |
When running tests with multiple parallel workers (as in C.I. and locally by default), SimpleCov was not correctly merging the coverage reports from each individual worker. As a result, our test coverage was being mis-reported as 63%. The issue dates back to commit 979437 which removed the parallel_tests gem, which SimpleCov detects automatically and works well alongside, replacing it with Rails 6's native test parallelization, which SimpleCov has known issues with. This commit adds a few lines of extra config on parallelization setup/teardown to fix SimpleCov's naming of the individual coverage reports, and to correctly surface the coverage output from each worker
aldavidson
force-pushed
the
fix-parallel-test-coverage
branch
from
1122eb7
to
b5d496d
Compare
Eep! I'm rusty :) amended now, thanks |
When running tests with multiple parallel workers (as in C.I. and locally by default), SimpleCov was not correctly merging the coverage reports from each individual worker. As a result, our test coverage was being mis-reported as 63%.
The issue dates back to commit 979437 which removed the parallel_tests gem, which SimpleCov detects automatically and works well alongside, replacing it with Rails 6's native test parallelization, which SimpleCov has known issues with.
This PR adds a few lines of extra config on parallelization setup/teardown to fix SimpleCov's naming of the individual coverage reports, and to correctly surface the coverage output from each worker
Before:
After: