formula-analytics: replace PyCall with subprocess - #23379
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR removes Homebrew’s in-process Python embedding (PyCall) from brew formula-analytics and replaces it with an out-of-process Python bridge script invoked via subprocess, with Python dependencies managed by uv (including lockfile support). It also updates CI and dependency automation to align with the new Python toolchain approach.
Changes:
- Replace PyCall-based InfluxDB querying with a stdin/stdout JSON protocol to a
uv-managed Python virtualenv and bridge script. - Remove the
pycallgem and associated Sorbet/Tapioca configuration and stubs. - Add
uvproject files (pyproject.toml,uv.lock,.python-version) and update CI (setup-uv) and Dependabot (uv ecosystem).
Reviewed changes
Copilot reviewed 18 out of 22 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| Library/Homebrew/vendor/bundle/bundler/setup.rb | Removes PyCall load paths from vendored bundler setup. |
| Library/Homebrew/test/support/helper/cmd/brew-verify-undefined.rb | Adds post-require undefined-constant guards for formula-analytics-related constants. |
| Library/Homebrew/test/dev-cmd/formula-analytics_spec.rb | Updates existing specs for the new query pathway and adds tests for the bridge protocol/error handling. |
| Library/Homebrew/sorbet/tapioca/require.rb | Stops special-casing/ignoring pycall during Tapioca requires. |
| Library/Homebrew/sorbet/tapioca/config.yml | Removes pycall from skipped gem RBI generation list. |
| Library/Homebrew/Gemfile.lock | Drops pycall from the Ruby dependency lockfile. |
| Library/Homebrew/Gemfile | Removes the optional :formula_analytics gem group that included pycall. |
| Library/Homebrew/formula-analytics/uv.lock | Adds a locked, hash-verified uv dependency set for the Python environment. |
| Library/Homebrew/formula-analytics/transform_analytics_to_counts.json | Removes an unused legacy Flux task file. |
| Library/Homebrew/formula-analytics/requirements.txt | Removes pip-compiled requirements in favor of uv.lock. |
| Library/Homebrew/formula-analytics/requirements.in | Removes the pip-compile input file. |
| Library/Homebrew/formula-analytics/pyproject.toml | Adds uv/PEP 621 project metadata and dependencies. |
| Library/Homebrew/formula-analytics/pycall-setup.rbi | Removes PyCall/InfluxDBClient3 Sorbet stubs no longer needed. |
| Library/Homebrew/formula-analytics/pycall-setup.rb | Removes PyCall-based import/setup code. |
| Library/Homebrew/formula-analytics/influxdb-query.py | Adds the Python bridge script implementing the stdin request / JSONL response protocol. |
| Library/Homebrew/formula-analytics/.python-version | Updates pinned Python version for the uv environment. |
| Library/Homebrew/dev-cmd/formula-analytics.rb | Switches formula-analytics to uv sync + subprocess bridge execution and updates --setup semantics. |
| Library/.rubocop.yml | Removes pycall-related RBI path exclusion. |
| completions/zsh/_brew | Updates --setup completion description to Python/uv semantics. |
| completions/fish/brew.fish | Updates --setup completion description to Python/uv semantics. |
| .github/workflows/tests.yml | Replaces actions/setup-python usage with astral-sh/setup-uv for formula-analytics CI. |
| .github/dependabot.yml | Switches dependency updates from pip to uv ecosystem for formula-analytics. |
Files not reviewed (1)
- Library/Homebrew/formula-analytics/pycall-setup.rbi: File type not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This was referenced Jul 31, 2026
MikeMcQuaid
force-pushed
the
replace-pycall-gem
branch
from
July 31, 2026 15:45
6465f69 to
87f7ba0
Compare
MikeMcQuaid
force-pushed
the
replace-pycall-gem
branch
from
July 31, 2026 19:31
87f7ba0 to
4653ee8
Compare
- `pycall` was Homebrew's only in-process Python embedding: a native gem that dlopens `libpython`, needed the `__PYVENV_LAUNCHER__` hack for macOS framework Pythons, hand-written Sorbet stubs and tapioca/RuboCop exclusions. - A pure Ruby HTTP port was rejected: InfluxDB Cloud Serverless only serves SQL over Arrow Flight gRPC (it has no `/api/v3/query_sql` endpoint) and the analytics schema keeps `package`, `tap_name` and `options` as fields, which InfluxQL cannot `GROUP BY`; Flux is deprecated on InfluxDB 3. Ruby Flight clients need heavier native dependencies than `pycall` itself. - Moving the whole query operation into Python was rejected: the JSON output depends on `Homebrew::EnvConfig` defaults, `MacOSVersion` pretty names and WSL suffix handling that would drift if duplicated outside Ruby. - `influxdb-query.py` runs from a `uv`-managed virtualenv with the same pinned `influxdb3-python` and speaks a documented protocol: a JSON request on stdin, JSON Lines rows on stdout and the token read from `HOMEBREW_INFLUXDB_TOKEN` so credentials stay off the command line. - Arrow record batches now stream via `mode="reader"` instead of being copied record-by-record across PyCall's FFI bridge. - Python dependencies moved from `pip-compile`d `requirements.txt` to `pyproject.toml` and a hash-verified `uv.lock` installed with `uv sync --frozen`: `uv` is a single dependency-free bottle installed 2.5x as often as versioned Python formulae, replaces the five bottles `python@3.13` needed and cold-installs the whole virtualenv faster than `pip` installed the packages alone, so `actions/setup-python` and dependabot's `pip` ecosystem are replaced by `astral-sh/setup-uv` and the `uv` ecosystem. - `uv` also provisions the interpreter: `.python-version` pins 3.14, the newest Python in Homebrew, and `requires-python = ">=3.13"` keeps future bumps to a one-line `.python-version` change. - `brew formula-analytics --setup` still prepares everything for offline runs and verifies the bridge import via the script's `--check` flag. - `brew verify-undefined` also guards `InfluxDBClient3` and `PyCall` after requiring `dev-cmd/formula-analytics`. - The TSM-era `transform_analytics_to_counts.json` Flux task was referenced by nothing so is removed.
p-linnane
force-pushed
the
replace-pycall-gem
branch
from
July 31, 2026 23:55
4653ee8 to
c2439e9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pycallwas Homebrew's only in-process Python embedding: a native gem that dlopenslibpython, needed the__PYVENV_LAUNCHER__hack for macOS framework Pythons, hand-written Sorbet stubs and tapioca/RuboCop exclusions./api/v3/query_sqlendpoint) and the analytics schema keepspackage,tap_nameandoptionsas fields, which InfluxQL cannotGROUP BY; Flux is deprecated on InfluxDB 3. Ruby Flight clients need heavier native dependencies thanpycallitself.Homebrew::EnvConfigdefaults,MacOSVersionpretty names and WSL suffix handling that would drift if duplicated outside Ruby.influxdb-query.pyruns from auv-managed virtualenv with the same pinnedinfluxdb3-pythonand speaks a documented protocol: a JSON request on stdin, JSON Lines rows on stdout and the token read fromHOMEBREW_INFLUXDB_TOKENso credentials stay off the command line.mode="reader"instead of being copied record-by-record across PyCall's FFI bridge.pip-compiledrequirements.txttopyproject.tomland a hash-verifieduv.lockinstalled withuv sync --frozen:uvis a single dependency-free bottle installed 2.5x as often as versioned Python formulae, replaces the five bottlespython@3.13needed and cold-installs the whole virtualenv faster thanpipinstalled the packages alone, soactions/setup-pythonand dependabot'spipecosystem are replaced byastral-sh/setup-uvand theuvecosystem.uvalso provisions the interpreter:.python-versionpins 3.14, the newest Python in Homebrew, andrequires-python = ">=3.13"keeps future bumps to a one-line.python-versionchange.brew formula-analytics --setupstill prepares everything for offline runs and verifies the bridge import via the script's--checkflag.brew verify-undefinedalso guardsInfluxDBClient3andPyCallafter requiringdev-cmd/formula-analytics.transform_analytics_to_counts.jsonFlux task was referenced by nothing so is removed.brewcommands to reproduce the bug?brew lgtm(style, typechecking and tests) locally?Claude Fable 5 max with local review and testing.