glib and friends: migrate to python@3.10 #87277
Conversation
carlocab
added
CI-long-timeout
I'm not sure I understand this question. But, here's a rough classification scheme:
Do you mean what happens if they are single-formula PRs? If they aren't single-formula PRs, they look like PRs like this one. I'm not sure this can get to a state where it's reasonable to merge this (at least, not while Python 3.9 is maintained):
If they are single-formula PRs that keep Python as a runtime dependency, the tap-syntax job will immediately fail with an error like (This is what I was referring to in point 1 above. "any of these formulae" could be a slight exaggeration -- I may have added some here by mistake. But nearly all of these formulae are connected via the dependency graph.)
I think it's also weird to always install something even when a user isn't going to use it (even if just indirectly), so I think we have to accept some amount of weirdness one way or another. (Unless you don't think this is weird. Maybe I can convince you that it is?)
I don't think it's clear that this degrades user experience, and if it does, it's not a serious degradation. Here are ways it actually improves it:
The second point above is an especially commonly cited pain point among users.
This is true, but there's an important difference: formulae typically need and link with openssl/icu4c/readline. To the extent that they are more like optional Python bindings rather than linked libraries like openssl, then I would say we should treat them the same way. I agree that it would be nice to have a way to deal with these more directly too, though.
I think "until recently" here is important. Python's dependency tree grows much faster than our CI capacity. When Python3.9 was added to Homebrew/core, 435 formulae formulae depended on Python 3.8 (74 of them build- or test-only). When Python3.10 was added, 587 formulae depended on Python3.9 (123 of them build- or test-only). (For reference, when |
|
Can these cases build with and/or be used at runtime with the system Python 3? That's not to say they should be but I think if they can be I'm a lot more 👍🏻 on making these
This is a wider problem that needs fixed. I've moaned on about this for years but we really need to get out of the habit of accepting flaky audits/tests and just delete them instead (or, if we have time [which we don't seem to] replace them).
I don't think this is weird if it's needed for any runtime functionality of what we build.
This all comes down to my first question, to me: can you actually use these Python bindings without installing our Python?
I second this. We need to find better solutions for the bigger problems here rather than just bypassing them in one-off ways like this (that may well come back to bite us).
Some of this seems like dependency bloat to me. We keep adding more and more dependencies to more and more formulae for consistency etc. even when we could be using a system SSL/Python/etc. Again, though, I agree that this is a problem that we need to solve. |
They can be built with system Python3 -- if the system provides one. Mojave and earlier didn't (which might be why all these formulae suddenly picked up dependencies on our Python3). Barring weird exceptions, these cases can also be used with any Python3 (system/Python.org/anaconda/etc) of the same minor version as the one that was used for the build. (This is actually what the Python audit I referenced above is meant to check for.) You could probably even get away with using them with a minor version that's one-off from the one that was used for the build, but that's asking for trouble.
I know about your opinion regarding flaky tests. I share your disdain for them, but I'm reluctant to just delete them as they can provide useful information. Maybe a label that allows us to skip non-essential tests when needed might be something we'd find useful? By "essential" I have in mind build, test, linkage, and offline audits, but I'm open to discussion about what this means exactly. (Failing this, I am open to entertaining just deleting these tests.)
I don't think it's correct to view all runtime functionality as equal. We already try to design
I'm a little surprised you agree with me here, since that's not the impression I got from your earlier responses. To make sure we're on the same page, what I meant was that if a formula has additional features that require extra dependencies, we should try to make those dependencies build- and test-only where possible, if we have to build them at all. I'd make exceptions for additional features that actually are of interest to most users of a formula: we should build those, and require the necessary dependencies at runtime.
Agreed. Dependency bloat is exactly the problem I'm trying to help solve here. I'm not saying that this should be the only thing we do, but I do think that this is one thing that can help and is a good idea. |
Gotcha. In that case I'm 👍🏻 on having it as a I think in general we should be more willing to use system versions of things, particularly on newer OSs.
Perhaps we should just skip these tests that can be flaky when doing dependent testing. If we're actually modifying a formula, though, it's a good time to actually fix the flaky issues. I agree improving the test is the best case for this but removing it is better than having a test that makes a multi-hour build turn red 25% of the time.
I agree but rather than shipping (on Mojave and below) broken functionality: we should just not ship it at all. Better to skip Python bindings entirely than ship them in a way that they don't work without new, manual, user intervention.
I agree with making things build or test only when they actually are i.e. there's no situation in which they are required at runtime. If they are useful/behaviour enhancing at runtime: cool, build or test it is. In the case of bindings (on Mojave and below): things are actually broken at runtime without them.
In summary: on Mojave and below I disagree, on Catalina and above I agree. Given we're no longer bottling for Mojave and below: I'm fine with whatever in this case. I do think it's worth talking about the general case, though. |
I think this will get us in trouble with the Linux side of the equation though. There we don't make any assumptions about things being installed. And however many dependencies we drop on macOS in favor of the system, they'll stay and block on Linux. |
Given appropriate
This, however, is likely to be a problem and, as we've added ARM and Linux builds, flaky tests/audits get increasingly likely to happen. |
|
I generally agree that if a software offers Python bindings and its not the primary use of it then a built-time dep makes sense and will not hamper UX (one doesn't attempt to use Python bindings without having Python installed anyway). We have a few formulae which already do this. I disagree however that it'll help much here in terms of the audit - the audit would (and should) still flag those kinds of version mismatches as we have no way of knowing whether those bindings are used or not in dependents.
System Python is stuck on 3.8 so you'll be restricted to that or our But note however there isn't a "system Python 3". There is one in the devtools and we do not require devtools to be installed when installing bottles. |
Yes, I was thinking of formulae that have multiple versions of GHC in their dep tree but only actually use one of them. I just remembered that we actually have an allowlist for this sort of thing.
We do require a CLT installation to install Python3, so anyone who might've installed Python already has the devtools installed. |
Gotcha, point taken then.
I guess I'm ambivalent either way given the conversation above. I just think if we ever get to the point of going "if you want to use Python bindings then run |
|
We've seen dependency-creep a lot in the past couples of years. Our general policy has been to build things with (in most cases) as many features as possible, and we carry a lot of heavy dependencies because of that. This is true of Python bindings, but it's also true of Qt: we have many more things that depend on Qt now. Even a simple install of I'm not necessarily opposed to it, but we probably want to be consistent. In context of other things, Python is actually a very lightweight dependency. Moreover, it is a quite common dependency, so many users probably have it anyway (like openssl). Mixing Python versions is a real pain, and Apple keeps saying “do not rely on system python”. Also, system Python is missing some really basic modules/packages. Therefore, I argue that the best course of action is probably to continue including python as dependency (probably even runtime dependency), and address the real problem: making testing smarter so we can do updates of things that have a lot of dependents. |
An aside but: I am opposed to it. Dependency bloat makes the package manager worse, in my opinion.
Agreed. |
| @@ -3,10 +3,10 @@ class Gnuradio < Formula | |||
|
|
|||
| desc "SDK for signal processing blocks to implement software radios" | |||
| homepage "https://gnuradio.org/" | |||
| url "https://github.com/gnuradio/gnuradio/archive/refs/tags/v3.9.3.0.tar.gz" | |||
| url "https://github.com/gnuradio/gnuradio/archive/refs/tags/v3.10.3.0.tar.gz" | |||
There was a problem hiding this comment.
Related to failure for:
brew fetch --retry gnuradio --build-bottle --force
Looks like accidental find-replace:
| url "https://github.com/gnuradio/gnuradio/archive/refs/tags/v3.10.3.0.tar.gz" | |
| url "https://github.com/gnuradio/gnuradio/archive/refs/tags/v3.9.3.0.tar.gz" |
Also, could optionally drop refs/tags part of URL.
| bin.env_script_all_files(libexec/"bin", PYTHONPATH: ENV["PYTHONPATH"]) | ||
| # This is already keg-only, so we can install everything into the prefix. | ||
| system "python3", *Language::Python.setup_install_args(prefix) |
There was a problem hiding this comment.
This is causing trouble with dependents as cython library is no longer automatically in PYTHONPATH. This would only work out-of-box if formula was linked (as it would then add cython libraries into default HOMEBREW_PREFIX/lib/python3.x).
Some options are to still re-add env variable for PYTHONPATH, manually set PYTHONPATH in each dependent, or remove keg-only and link it.
There was a problem hiding this comment.
Good catch. I'm not opposed to making this not-keg-only. I think this being keg-only just comes from our past policy of preferring users to pip install everything they can install with pip. We've relaxed that to allowing Python modules that expose a CLI, and that includes cython.
There was a problem hiding this comment.
I wonder if there is a minor risk of de-sync'ing formulae if a user runs pip install -U cython or pip install cython==<version>.
Though, that would be a user issue based on Homebrew's stance for python formulae.
Anyway, I'm also fine with just linking it for an easier-to-manage formula.
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
We're migrating a number of protobuf dependents on Python 3.10 in Homebrew#87277. We could migrate protobuf fully to Python 3.10 there too, but I'd like to avoid pulling in protobuf's dependency tree into the CI run there.
We're migrating a number of protobuf dependents on Python 3.10 in #87277. We could migrate protobuf fully to Python 3.10 there too, but I'd like to avoid pulling in protobuf's dependency tree into the CI run there. Closes #87287. Signed-off-by: BrewTestBot <1589480+BrewTestBot@users.noreply.github.com>
This is needed for Homebrew#94505. The conflict comes from `freedink` depending on `python@3.10` and `python@3.9` through `cxxtest` and `glib` respectively. `cxxtest` is installed into a virtualenv in `libexec`, so it's unlikely that `freedink` actually makes use of anything there since it ships no Python code itself. An alternative to this is to migrate `glib` to `python@3.10`, which will take quite a while (see Homebrew#87277), or migrate `cxxtest` back to `python@3.9`, which is unnecessary. I've tested `freedink` locally and it seems to work fine despite the mixed recursive dependency. See also Homebrew#65831.
This is needed for #94505. The conflict comes from `freedink` depending on `python@3.10` and `python@3.9` through `cxxtest` and `glib` respectively. `cxxtest` is installed into a virtualenv in `libexec`, so it's unlikely that `freedink` actually makes use of anything there since it ships no Python code itself. An alternative to this is to migrate `glib` to `python@3.10`, which will take quite a while (see #87277), or migrate `cxxtest` back to `python@3.9`, which is unnecessary. I've tested `freedink` locally and it seems to work fine despite the mixed recursive dependency. See also #65831. Closes #94505. Signed-off-by: BrewTestBot <1589480+BrewTestBot@users.noreply.github.com>
Let's hope I don't need to include more in this PR...