Skip to content

Fix SimpleCov coverage for parallel runs - #23358

Merged
MikeMcQuaid merged 1 commit into
mainfrom
fix-simplecov-parallel-coverage
Jul 30, 2026
Merged

Fix SimpleCov coverage for parallel runs#23358
MikeMcQuaid merged 1 commit into
mainfrom
fix-simplecov-parallel-coverage

Conversation

@samford

@samford samford commented Jul 29, 2026

Copy link
Copy Markdown
Member

  • Have you followed our Contributing guidelines?
  • Have you checked for other open Pull Requests for the same change?
  • Have you explained what your changes do? Performance claims (e.g. "this is faster") must include Hyperfine benchmarks.
  • Have you explained why you'd like these changes included, not just what they do?
  • For bug fixes, have you given step-by-step brew commands to reproduce the bug?
  • Have you written new tests (excluding integration tests)? Here's an example.
  • Have you successfully run brew lgtm (style, typechecking and tests) locally?

  • I did not use AI/LLM to create this PR, or I disclosed the tool/model below and reviewed its output; I did not attribute commits to AI and will answer maintainer questions and review comments myself without AI/LLM.

I used Claude Opus 5 (High) to identify the issue and implement the fixes. I reviewed the changes (having read the upstream SimpleCov documentation and changelog), made some tweaks, and compared brew tests --coverage output to confirm the fix (comparing the output for 0.22.0, the main branch, and this PR branch).


We recently updated SimpleCov from 0.22.0 to 1.0.1 but 1.0 involves some changes to how coverage data is merged/finalized when run in parallel and our preexisting setup doesn't quite work as expected. We addressed API deprecations when upgrading and brew tests --coverage works but it produces incomplete coverage data (i.e., the percentages are much lower than expected).

This appears to be due to a couple of issues:

  • In SimpleCov 1.0, finalize_merge false causes SimpleCov.result to return early, so workers will return their result without waiting for and merging results from other workers. We use it explicitly in .simplecov but it would also be set by default for parallel runs when a custom coverage_dir is set, as in our case. I think the expectation is that we would use SimpleCov.collate but our logic in spec_helper.rb doesn't.
  • We use result.format! if ParallelTests.last_process? in the SimpleCov.at_exit logic in spec_helper.rb but SimpleCov 1.0 switched to using first_process? (to align with parallel_tests), so that no longer works as expected.

This addresses the issue by using finalize_merge SimpleCov.final_result_process? and removing the now-unnecessary SimpleCov.at_exit logic in spec_helper.rb, as the default behavior in 1.0 already does what we want. finalize_merge true technically works but it's notably slower for no added benefit (i.e., only one worker needs to merge and format the results).

This also removes the parallel = false if args.coverage? && files.length < Hardware::CPU.cores workaround in dev-cmd/tests.rb, as the related issues have been resolved in 1.0.


I tested this by running brew tests --coverage using the commit before the SimpleCov 1.0 upgrade (09ad50f, which uses 0.22.0) and comparing that to the output on main (using 1.0.1). After implementing these changes, I ran it again and compared the previous 0.22.0 output to the output from this PR branch and confirmed that the coverage was generally what we would expect. I noticed a few small differences in the coverage but that may be attributable to how SimpleCov behaves in 1.0 (though that's something we can address in a follow-up PR, if needed).

We recently updated SimpleCov from 0.22.0 to 1.0.1 but 1.0 involves
some changes to how coverage data is merged/finalized when run in
parallel and our preexisting setup doesn't quite work as expected. We
addressed API deprecations when upgrading and `brew tests --coverage`
works but it produces incomplete coverage data (i.e., the percentages
are much lower than expected).

This appears to be due to a couple of issues:

* In SimpleCov 1.0, `finalize_merge false` causes `SimpleCov.result`
to return early, so workers will return their result without waiting
for and merging results from other workers. We use it explicitly in
`.simplecov` but it would also be set by default for parallel runs
when a custom `coverage_dir` is set, as in our case. I think the
expectation is that we would use `SimpleCov.collate` but our logic in
`spec_helper.rb` doesn't.
* We use `result.format! if ParallelTests.last_process?` in the
`SimpleCov.at_exit` logic in `spec_helper.rb` but SimpleCov 1.0
switched to using `first_process?` (to align with `parallel_tests`),
so that no longer works as expected.

This addresses the issue by using `finalize_merge
SimpleCov.final_result_process?` and removing the now-unnecessary
`SimpleCov.at_exit` logic in `spec_helper.rb`, as the default
behavior in 1.0 already does what we want. `finalize_merge true`
technically works but it's notably slower for no added benefit (i.e.,
only one worker needs to merge and format the results).

This also removes the `parallel = false if args.coverage? &&
files.length < Hardware::CPU.cores` workaround in `dev-cmd/tests.rb`,
as the related issues have been resolved in 1.0.
Copilot AI review requested due to automatic review settings July 29, 2026 23:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request updates Homebrew’s test coverage setup to work correctly with SimpleCov 1.0.x when RSpec tests are executed in parallel, ensuring coverage results are merged/finalized by the appropriate process and removing no-longer-needed workaround logic.

Changes:

  • Adjust SimpleCov configuration to set finalize_merge dynamically using SimpleCov.final_result_process? for parallel runs.
  • Remove custom SimpleCov.at_exit handling from test/spec_helper.rb that attempted to gate formatting to a single parallel worker.
  • Remove the --coverage parallel-disable workaround in dev-cmd/tests.rb for small test sets.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
Library/Homebrew/.simplecov Updates SimpleCov parallel merge finalization logic for correct coverage aggregation in 1.0.x.
Library/Homebrew/test/spec_helper.rb Removes now-unnecessary parallel-only SimpleCov.at_exit formatting guard.
Library/Homebrew/dev-cmd/tests.rb Removes coverage-related parallel execution workaround in the brew tests command.
Comments suppressed due to low confidence (1)

Library/Homebrew/dev-cmd/tests.rb:106

  • This change removes the previous guard that forced parallel = false for coverage runs with fewer spec files than CPU cores. Given this is a behavior change in brew tests (affecting whether we invoke parallel_rspec vs rspec under --coverage), it would be good to add/adjust an RSpec example in test/dev-cmd/tests_spec.rb that exercises #run with --coverage and a small file set and asserts we still run parallel_rspec (or, if intended otherwise, codifies the new expected behavior).
          parallel_rspec_log_name = "parallel_runtime_rspec"
          parallel_rspec_log_name = "#{parallel_rspec_log_name}.generic" if args.generic?
          parallel_rspec_log_name = "#{parallel_rspec_log_name}.online" if args.online?
          parallel_rspec_log_name = "#{parallel_rspec_log_name}.log"

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@p-linnane p-linnane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @samford!

@MikeMcQuaid MikeMcQuaid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, love a fix that's mostly deleting!

@MikeMcQuaid
MikeMcQuaid added this pull request to the merge queue Jul 30, 2026
Merged via the queue into main with commit 2eaef66 Jul 30, 2026
46 checks passed
@MikeMcQuaid
MikeMcQuaid deleted the fix-simplecov-parallel-coverage branch July 30, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants