Comparison to python-build-standalone

[rickeylev]

Hi 👋,

Given that python-build-standalone exists, I think it'd be good to clarify in what ways this will be different/similar. The two particular questions I think should be answered are:

1. As a user, how do I decide which to use?
2. As a user, if I've decided to switch to it, what caveats/gotchas/differences should I be aware of?

[emmatyping]

It's hard to answer these questions exactly because we haven't solidified on what the new builds will look like, but a few differences could be:

• PBS can build versions for newer architectures and distribute multiple versions, I expect we will build one distribution per arch with a conservative baseline
• PBS builds against libedit, I am hopeful that prebuilt-cpython will link against readline for compatibility, but that is a can of worms :)
• prebuilt-cpython can be signed by the official Python signing keys and that will likely provide some security assurances to some users, especially in enterprise environments
• PBS may support older distros for newer Python releases? This one is still unclear

That's what I can think of off the top of my head

[brettcannon]

This aligns with my thinking, but there's also an ambiguity in the question:

As a user

What sort of user are we talking about here? For instance, for general users, they shouldn't care as I suspect it will be tools that do the downloading for them. Otherwise I hope the differences are minimal enough that the common user won't care.

[geofft]

My personal opinion as one of the people who works on python-build-standalone: I think it would be a big success if these ended up being exactly the same thing, and I think that would be a good goal to get to. There are a number of ways you could imagine this working organizationally.

I also think any step in that direction would also be a big success, e.g., using the same code to drive the build but with the CPython team owning the build process. On the technical side, the build scripts are very intentionally open source (MPL) and so nothing stands in the way of using the existing project as a starting point. At the least, I think we should aim for getting all the patches for relocatability upstream.

I suspect one practical difference is that python-build-standalone currently has slightly less conservative build options enabled (e.g., using BOLT, turning on the JIT in yes-off mode) and I'm not sure if something offered by upstream CPython will want to be more conservative about what's supported in an official build. Astral also has committed / non-volunteer time on the project which makes it easier to be more ambitious about what options are enabled.

Also worth noting: we distribute the entire stack of libraries that Python depends on up to the system C library, namely OpenSSL, Tcl/Tk for tkinter and therefore libX11, ncurses, etc. This is kind of annoying to maintain. A reasonable approach for an official build would be to do a little less of this and require more dependencies from the system.

python-build-standalone came out of a use case for building single-file applications written in Python for PyOxidizer, so there's a handful of design decisions like trying to link all compiled modules in the standard library into the executable instead of having them be separate loadable modules on the filesystem. That use case is certainly not the primary one and we're shifting away a bit from accommodating it (e.g., we had to spin out _tkinter into a separate file for compatibility with matplotlib etc.). Presumably such a goal is completely out of scope for this project. But again, that doesn't mean the two projects can't converge; there could be an optional single-library build that isn't part of the official release artifacts distributed by CPython.

[geofft]

Another option to keep it out of github.com/python/ is to leave the repo where it is and treat it as an external build tool, i.e., the release scripts would just clone that repo and invoke the build with particular arguments. I'm happy to open a PR for that if it's interesting (even as a draft / proof of concept), but I wonder if that's actually helpful or not to this goal. That design would mean that the code would remain under astral-sh/, which you may not want, and also, given your comment, you might not feel comfortable contributing to an MPL-licensed repo (regardless of where it is) and to CPython at the same time.

If we are able to resolve the licensing issue (either by the PSF deciding to accepting the license or by relicensing), do you think the python-build-standalone code works as a starting point? If so, then IMO the first step is we should figure out relicensing :)

(Is the text at https://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq#What_if_I_want_to_contribute_my_code_to_the_PSF.3F still accurate that the preferred license for contributions to the PSF is Academic Free or Apache?)

[geofft]

I think "exact" might be too much, but that's based on comments that I vaguely remember @indygreg making in terms of decisions to facilitate PyOxidizer which might not make sense to have upstreamed.

See astral-sh/python-build-standalone#227.

From my perspective, the primary audience of the release artifacts of python-build-standalone is the same as the audience of this project, namely, regular Python users who want a normal-looking Python that they can install in their home directory. Relocatability / self-contained-ness is useful because it means you can install it into a path that isn't known at build time and because you don't need sudo access to install dependencies, but that's just a subset of the goals of the single-file-embeddable-library use case. I'm happy to keep the build support for the single-file-embeddable-library approach working as far as possible just like there are other configs in the repo that aren't built but are still around (static musl builds that cannot use compiled extension libraries because they cannot dlopen, some 32-bit configs, some iOS configs, etc.), but certainly for the "install-only" artifacts, I think they're targeting your use case better than the PyOxidizer-style use case.

[zooba]

The number of contributors to the MPL code in PBS is exceptionally minimal. I likely authored 99% of code until Astral took ownership.

This basically resolves the concern. If it can either be relicensed in place (technically a bit easier) or recontributed (but it's not clear where to put it yet) under Apache 2.0, then it'll be easy.

Though the way you say "to the MPL code" suggests it's already a mixed license repository? How much is actually under MPL?

the preferred license for contributions to the PSF is Academic Free or Apache

More strongly towards Apache these days, I think, but presumably you've signed the CPython CLA at some stage? So contributing under that CLA (and thereby asserting that you have the right to contribute under that CLA) is all that really matters.

Though again, we don't have anywhere to put it yet, which is why relicensing in place is a bit easier right now.

[brettcannon]

do you think the python-build-standalone code works as a starting point?

I have no idea. First thing I wanted to tackle was patches to CPython getting upstreamed, then we can worry about scripting the build.

[indygreg]

Though the way you say "to the MPL code" suggests it's already a mixed license repository? How much is actually under MPL?

PBS may be 100% MPL. I can't recall. When doing PyOxidizer, I intentionally made the build time stuff MPL and the run time stuff more permissive so the magic build time pieces would remain OSS without creating licensing issues for downstream applications/users. If I followed this principle, PBS should be ~100% MPL.

Various patches to CPython may not be explicitly licensed, which creates some licensing ambiguity. This could be fixed easily enough.