Skip to content

perf: add Utils.parallel_map, use it in tap-info and services list#22992

Merged
MikeMcQuaid merged 1 commit into
mainfrom
parallel-map-helper
Jul 10, 2026
Merged

perf: add Utils.parallel_map, use it in tap-info and services list#22992
MikeMcQuaid merged 1 commit into
mainfrom
parallel-map-helper

Conversation

@dduugg

@dduugg dduugg commented Jul 6, 2026

Copy link
Copy Markdown
Member

Summary

In #22975 the per-tap Tap#to_hash calls were made concurrent with one plain Thread per tap, and review suggested extracting a single block helper if the approach got reused. This PR adds that helper, applies it to the serial GitHub API loop in bare brew tap-info and retrofits the inline-thread call sites from #22975 and #22989.

Changes

  • Utils.parallel_map(items, &block): one thread per item, results in input order, first exception re-raised by Thread#value (with report_on_exception disabled so it isn't also printed to stderr). The docstring notes it is only worthwhile for blocking waits, since the GVL serializes Ruby execution.
  • brew tap-info (no arguments): the statistics loop called Tap#private? serially, one GitHub API request per non-core tap, plus per-tap file counting.
  • brew tap-info --installed --json: retrofitted to use the helper; behavior and performance unchanged from perf/tap-info: generate tap hashes concurrently (up to ~77% faster) #22975.
  • SystemConfig.dump_verbose_config: retrofitted the inline-thread section rendering from perf/config: gather system configuration concurrently (~31% faster) #22989 (merged after this PR was opened); behavior and performance unchanged, brew config output verified identical.

Per review, an earlier brew services list conversion was dropped: its benchmark was within noise at typical service counts.

Output is byte-identical before and after.

Benchmarks (Hyperfine, 10 runs each, warmup 2, HOMEBREW_SORBET_RUNTIME unset, 9 taps of which 7 need a GitHub API call):

Command Before After Change
brew tap-info 3.295 s ± 0.160 s 1.602 s ± 0.069 s ~51% decrease

  • Have you followed our Contributing guidelines?
  • Have you checked for other open Pull Requests for the same change?
  • Have you explained what your changes do? Performance claims (e.g. "this is faster") must include Hyperfine benchmarks.
  • Have you explained why you'd like these changes included, not just what they do?
  • For bug fixes, have you given step-by-step brew commands to reproduce the bug?
  • Have you written new tests (excluding integration tests)? Here's an example.
  • Have you successfully run brew lgtm (style, typechecking and tests) locally?

  • AI was used to generate or assist with generating this PR.

Claude Code was used to profile the commands, implement the helper and call sites and write the helper specs (ordering under staggered sleeps, exception propagation). Verified by byte-identical before/after output, interleaved Hyperfine benchmarks on clean trees and running brew lgtm locally.


@dduugg dduugg force-pushed the parallel-map-helper branch from 813771e to abdd9b8 Compare July 6, 2026 20:06

@MikeMcQuaid MikeMcQuaid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Think this is worth it for tap-info but not for services based on your benchmarks.

Also would be fine if Concurrent is "better" if you wanted to use that inside the helper instead (but could be follow-up)

@dduugg dduugg force-pushed the parallel-map-helper branch from abdd9b8 to a0cf4f8 Compare July 10, 2026 00:25
@dduugg

dduugg commented Jul 10, 2026

Copy link
Copy Markdown
Member Author

Thanks @MikeMcQuaid! Dropped the services list conversion, so the PR is now the helper, the bare brew tap-info stats loop (~51% decrease) and the #22975 retrofit. Kept plain Threads inside the helper for now: I benchmarked a Concurrent::FixedThreadPool version against plain threads earlier and they were identical here (1.25 s vs 1.23 s), since the work is all subprocess/network waits. Happy to switch the internals in a follow-up if a bounded pool becomes useful.

@dduugg dduugg force-pushed the parallel-map-helper branch from a0cf4f8 to 8bee91d Compare July 10, 2026 00:31
@dduugg

dduugg commented Jul 10, 2026

Copy link
Copy Markdown
Member Author

Since the last comment: #22989 merged, so its inline-thread section rendering in SystemConfig.dump_verbose_config has also been retrofitted to Utils.parallel_map, giving the helper a third call site. brew config output verified identical before and after, with no benchmark change.

@dduugg dduugg marked this pull request as ready for review July 10, 2026 00:52
Copilot AI review requested due to automatic review settings July 10, 2026 00:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a reusable Utils.parallel_map helper to run per-item work concurrently (one thread per item) and applies it to places that were previously doing ad-hoc threading or serial GitHub API waits, notably improving brew tap-info performance while preserving output order.

Changes:

  • Add Utils.parallel_map(items, &block) to run blocking work concurrently and return results in input order.
  • Use Utils.parallel_map in brew tap-info (no-args stats and --json) to overlap per-tap GitHub API calls and other I/O.
  • Replace inline thread rendering in SystemConfig.dump_verbose_config with the new helper.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
Library/Homebrew/utils.rb Adds Utils.parallel_map threading helper and documentation.
Library/Homebrew/test/utils_spec.rb Adds tests for Utils.parallel_map behavior.
Library/Homebrew/system_config.rb Refactors verbose config section rendering to use Utils.parallel_map.
Library/Homebrew/cmd/tap-info.rb Parallelizes per-tap stats collection and retrofits JSON generation to use Utils.parallel_map.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread Library/Homebrew/test/utils_spec.rb Outdated
Comment thread Library/Homebrew/utils.rb Outdated
@dduugg dduugg force-pushed the parallel-map-helper branch from 8bee91d to e7277d9 Compare July 10, 2026 01:09

@MikeMcQuaid MikeMcQuaid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@MikeMcQuaid MikeMcQuaid added this pull request to the merge queue Jul 10, 2026
Merged via the queue into main with commit 46c9bd3 Jul 10, 2026
41 checks passed
@MikeMcQuaid MikeMcQuaid deleted the parallel-map-helper branch July 10, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants