Skip to content

formula-analytics: replace PyCall with subprocess - #23379

Merged
p-linnane merged 1 commit into
mainfrom
replace-pycall-gem
Aug 1, 2026
Merged

formula-analytics: replace PyCall with subprocess#23379
p-linnane merged 1 commit into
mainfrom
replace-pycall-gem

Conversation

@MikeMcQuaid

Copy link
Copy Markdown
Member
  • pycall was Homebrew's only in-process Python embedding: a native gem that dlopens libpython, needed the __PYVENV_LAUNCHER__ hack for macOS framework Pythons, hand-written Sorbet stubs and tapioca/RuboCop exclusions.
  • A pure Ruby HTTP port was rejected: InfluxDB Cloud Serverless only serves SQL over Arrow Flight gRPC (it has no /api/v3/query_sql endpoint) and the analytics schema keeps package, tap_name and options as fields, which InfluxQL cannot GROUP BY; Flux is deprecated on InfluxDB 3. Ruby Flight clients need heavier native dependencies than pycall itself.
  • Moving the whole query operation into Python was rejected: the JSON output depends on Homebrew::EnvConfig defaults, MacOSVersion pretty names and WSL suffix handling that would drift if duplicated outside Ruby.
  • influxdb-query.py runs from a uv-managed virtualenv with the same pinned influxdb3-python and speaks a documented protocol: a JSON request on stdin, JSON Lines rows on stdout and the token read from HOMEBREW_INFLUXDB_TOKEN so credentials stay off the command line.
  • Arrow record batches now stream via mode="reader" instead of being copied record-by-record across PyCall's FFI bridge.
  • Python dependencies moved from pip-compiled requirements.txt to pyproject.toml and a hash-verified uv.lock installed with uv sync --frozen: uv is a single dependency-free bottle installed 2.5x as often as versioned Python formulae, replaces the five bottles python@3.13 needed and cold-installs the whole virtualenv faster than pip installed the packages alone, so actions/setup-python and dependabot's pip ecosystem are replaced by astral-sh/setup-uv and the uv ecosystem.
  • uv also provisions the interpreter: .python-version pins 3.14, the newest Python in Homebrew, and requires-python = ">=3.13" keeps future bumps to a one-line .python-version change.
  • brew formula-analytics --setup still prepares everything for offline runs and verifies the bridge import via the script's --check flag.
  • brew verify-undefined also guards InfluxDBClient3 and PyCall after requiring dev-cmd/formula-analytics.
  • The TSM-era transform_analytics_to_counts.json Flux task was referenced by nothing so is removed.

  • Have you followed our Contributing guidelines?
  • Have you checked for other open Pull Requests for the same change?
  • Have you explained what your changes do? Performance claims (e.g. "this is faster") must include Hyperfine benchmarks.
  • Have you explained why you'd like these changes included, not just what they do?
  • For bug fixes, have you given step-by-step brew commands to reproduce the bug?
  • Have you written new tests (excluding integration tests)? Here's an example.
  • Have you successfully run brew lgtm (style, typechecking and tests) locally?

  • I did not use AI/LLM to create this PR, or I disclosed the tool/model below and reviewed its output; I did not attribute commits to AI and will answer maintainer questions and review comments myself without AI/LLM.

Claude Fable 5 max with local review and testing.


Copilot AI review requested due to automatic review settings July 31, 2026 15:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes Homebrew’s in-process Python embedding (PyCall) from brew formula-analytics and replaces it with an out-of-process Python bridge script invoked via subprocess, with Python dependencies managed by uv (including lockfile support). It also updates CI and dependency automation to align with the new Python toolchain approach.

Changes:

  • Replace PyCall-based InfluxDB querying with a stdin/stdout JSON protocol to a uv-managed Python virtualenv and bridge script.
  • Remove the pycall gem and associated Sorbet/Tapioca configuration and stubs.
  • Add uv project files (pyproject.toml, uv.lock, .python-version) and update CI (setup-uv) and Dependabot (uv ecosystem).

Reviewed changes

Copilot reviewed 18 out of 22 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
Library/Homebrew/vendor/bundle/bundler/setup.rb Removes PyCall load paths from vendored bundler setup.
Library/Homebrew/test/support/helper/cmd/brew-verify-undefined.rb Adds post-require undefined-constant guards for formula-analytics-related constants.
Library/Homebrew/test/dev-cmd/formula-analytics_spec.rb Updates existing specs for the new query pathway and adds tests for the bridge protocol/error handling.
Library/Homebrew/sorbet/tapioca/require.rb Stops special-casing/ignoring pycall during Tapioca requires.
Library/Homebrew/sorbet/tapioca/config.yml Removes pycall from skipped gem RBI generation list.
Library/Homebrew/Gemfile.lock Drops pycall from the Ruby dependency lockfile.
Library/Homebrew/Gemfile Removes the optional :formula_analytics gem group that included pycall.
Library/Homebrew/formula-analytics/uv.lock Adds a locked, hash-verified uv dependency set for the Python environment.
Library/Homebrew/formula-analytics/transform_analytics_to_counts.json Removes an unused legacy Flux task file.
Library/Homebrew/formula-analytics/requirements.txt Removes pip-compiled requirements in favor of uv.lock.
Library/Homebrew/formula-analytics/requirements.in Removes the pip-compile input file.
Library/Homebrew/formula-analytics/pyproject.toml Adds uv/PEP 621 project metadata and dependencies.
Library/Homebrew/formula-analytics/pycall-setup.rbi Removes PyCall/InfluxDBClient3 Sorbet stubs no longer needed.
Library/Homebrew/formula-analytics/pycall-setup.rb Removes PyCall-based import/setup code.
Library/Homebrew/formula-analytics/influxdb-query.py Adds the Python bridge script implementing the stdin request / JSONL response protocol.
Library/Homebrew/formula-analytics/.python-version Updates pinned Python version for the uv environment.
Library/Homebrew/dev-cmd/formula-analytics.rb Switches formula-analytics to uv sync + subprocess bridge execution and updates --setup semantics.
Library/.rubocop.yml Removes pycall-related RBI path exclusion.
completions/zsh/_brew Updates --setup completion description to Python/uv semantics.
completions/fish/brew.fish Updates --setup completion description to Python/uv semantics.
.github/workflows/tests.yml Replaces actions/setup-python usage with astral-sh/setup-uv for formula-analytics CI.
.github/dependabot.yml Switches dependency updates from pip to uv ecosystem for formula-analytics.
Files not reviewed (1)
  • Library/Homebrew/formula-analytics/pycall-setup.rbi: File type not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread Library/Homebrew/formula-analytics/influxdb-query.py

@p-linnane p-linnane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice!

- `pycall` was Homebrew's only in-process Python embedding: a native
  gem that dlopens `libpython`, needed the `__PYVENV_LAUNCHER__` hack
  for macOS framework Pythons, hand-written Sorbet stubs and
  tapioca/RuboCop exclusions.
- A pure Ruby HTTP port was rejected: InfluxDB Cloud Serverless only
  serves SQL over Arrow Flight gRPC (it has no `/api/v3/query_sql`
  endpoint) and the analytics schema keeps `package`, `tap_name` and
  `options` as fields, which InfluxQL cannot `GROUP BY`; Flux is
  deprecated on InfluxDB 3. Ruby Flight clients need heavier native
  dependencies than `pycall` itself.
- Moving the whole query operation into Python was rejected: the JSON
  output depends on `Homebrew::EnvConfig` defaults, `MacOSVersion`
  pretty names and WSL suffix handling that would drift if duplicated
  outside Ruby.
- `influxdb-query.py` runs from a `uv`-managed virtualenv with the
  same pinned `influxdb3-python` and speaks a documented protocol: a
  JSON request on stdin, JSON Lines rows on stdout and the token read
  from `HOMEBREW_INFLUXDB_TOKEN` so credentials stay off the command
  line.
- Arrow record batches now stream via `mode="reader"` instead of
  being copied record-by-record across PyCall's FFI bridge.
- Python dependencies moved from `pip-compile`d `requirements.txt` to
  `pyproject.toml` and a hash-verified `uv.lock` installed with
  `uv sync --frozen`: `uv` is a single dependency-free bottle
  installed 2.5x as often as versioned Python formulae, replaces the
  five bottles `python@3.13` needed and cold-installs the whole
  virtualenv faster than `pip` installed the packages alone, so
  `actions/setup-python` and dependabot's `pip` ecosystem are
  replaced by `astral-sh/setup-uv` and the `uv` ecosystem.
- `uv` also provisions the interpreter: `.python-version` pins 3.14,
  the newest Python in Homebrew, and `requires-python = ">=3.13"`
  keeps future bumps to a one-line `.python-version` change.
- `brew formula-analytics --setup` still prepares everything for
  offline runs and verifies the bridge import via the script's
  `--check` flag.
- `brew verify-undefined` also guards `InfluxDBClient3` and `PyCall`
  after requiring `dev-cmd/formula-analytics`.
- The TSM-era `transform_analytics_to_counts.json` Flux task was
  referenced by nothing so is removed.
@p-linnane
p-linnane force-pushed the replace-pycall-gem branch from 4653ee8 to c2439e9 Compare July 31, 2026 23:55
@p-linnane
p-linnane added this pull request to the merge queue Aug 1, 2026
Merged via the queue into main with commit 9ba8223 Aug 1, 2026
49 checks passed
@p-linnane
p-linnane deleted the replace-pycall-gem branch August 1, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants