utils/pypi: exclude deps of excluded packages #15896
Conversation
|
I felt like it broke the brew. Merge problem fatal: couldn't find remote ref refs/heads/master Error: Some taps failed to update! The following taps can not read their remote branches: dart-lang/dart This is happening because the remote branch was renamed or deleted. |
| # Resolve the dependency tree of excluded packages to prune the above | ||
| exclude_packages.delete_if { |package| found_packages.exclude? package } | ||
| ohai "Retrieving PyPI dependencies for excluded \"#{exclude_packages.join(" ")}\"..." if !print_only && !silent | ||
| exclude_packages = pip_report(exclude_packages) + [Package.new(main_package.name)] |
There was a problem hiding this comment.
This might incorrectly exclude common deps shared by the main package and some of the exclude_packages, I think.
Take Homebrew/homebrew-core#140275 as an example (see Homebrew/homebrew-core@f4c55ea). Here, moto, the main package, depends on boto3 and botocore. One of its exclude_packages, cfn-lint, also depends on boto3 and botocore. IIUC, with this logic, boto3 and botocore are incorrectly removed.
There was a problem hiding this comment.
I think that's what we want, since in python there's only a single version of a dep shared by the whole venv (so no need to install it twice).
That formula actually has a bug, its depending on cfn-lint without it being importable so neither is boto3 (it needs a .pth file to find it)
$(brew --prefix)/opt/moto/libexec/bin/python -c 'import cfnlint'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cfnlint'
|
Hmm.... I've setup a tap for company-internal tools. (I realize that isn't the goal of homebrew so I'm kinda on my own.) I have a formula for a python CLI which depends on another company-internal python library. That company-internal python library is github private repo. I previously had done
In the formula for resource "private-python-lib" do
url "ssh://git@github.com/mycompany/private-python-lib.git",
using: :git,
tag: "v1.2.1"
endI think a fix for this might be to have the "look up dependencies of excluded packages" logic skip non-pypi packages (when [edit]OK I attempted that fix I suggested. A relatively-simple approach didn't work, I suspect because the mix/confusion of simple package names (e.g. Given that I'm trying to use homebrew in a way outside of its goals, I think I should figure out my own solution to this (and not try to change |
brew stylewith your changes locally?brew typecheckwith your changes locally?brew testswith your changes locally?This teaches
brew update-python-resourcesto also ignore the dependencies of excluded packages (since we exclude packages provided instead by formulae, which of course already contain these required deps).In core, our
exclude_packagesinpypi_formula_mappings.jsonstarted out with simple packages without dependencies likesixwhere this distinction didn't matter. But now, our use has grown to exclude packages with their own dependencies (cryptography,keyring, andvirtualenv). So this change reduces the manual maintainence of our formula mappings.