-
Notifications
You must be signed in to change notification settings - Fork 33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Re-think no-binary support for build dependencies (bootstrapping problem) #342
Comments
@brainwane @ewdurbin @di @dstufft ^ for the Packaging-WG "Fundable Packaging Improvements" page. |
I'm interested in adding this to the list of fundable packaging improvements, but I need help with wording. Try answering each of these questions in about 20-200 words each:
|
Anyone who wants to add this to the list of fundable packaging improvements can now submit a pull request on https://github.com/psf/fundable-packaging-improvements . |
This issue is more general than just wheel and affects a number of packages beyond setuptools and wheel. In pypa/setuptools#980, I learned of the issue in build tools. If build tools have dependencies, and those dependencies use the build tool, it's not possible to build the dependencies from source. For example,
An attempt to build/install Setuptools or Appdirs from source rather than wheels (as many platforms prefer to do) will fail. Setuptools has worked around this issue by vendoring all of its dependencies, although it has a stated goal to stop vendoring (pypa/setuptools#2825). However, in python/importlib_metadata#392 today, I've learned of even more places where these circular dependencies can manifest. In this case:
So unless I've probably stated before, I wish for libraries like These new cases lead me to believe the problem is not solveable by downstream packagers except by forcing all build tools (and their plugins) to vendor their dependencies. I'd like to avoid each of these projects needing to come up with a bespoke hack to work around this issue and eliminate this footgun that will reduce adoption due to these bootstrapping issues. |
The situation is in any context where in a project, any of the PEP 518
First, it's not obvious what needs to be implemented. This project still needs work to explore the problem space and propose options to be considered by the PyPA. It's going to require coordination across many projects and may require a PEP. A rough sketch of some options to consider:
It would solve the class of problems where errors occur when users attempt to build from sources but dependencies exist in the build dependencies. It would mean that build tools like |
Based on the criteria in the readme there, I don't believe this project is yet ready for a fundable project. There is not yet consensus or an accepted PEP. To the contrary, the initial feedback has been negative, meaning there's probably a need to develop that consensus first. Maybe the additional use case described above will help nudge us in that direction. In the meantime, I'm slightly inclined not to workaround the issues for users, thus masking the root cause. |
I think there's a good fundable project in here, but it's a research project. I think the problem is that we are getting feedback that people want to be able to "build everything from source", but that causes an issue because bootstrapping involves circular dependencies. This sounds pretty much identical to the issues of building a C compiler, so we could probably learn a lot from how people solved that problem. Also, presumably the people wanting to build Python packages from scratch stop at some point - do they need to build the Python interpreter itself from sources, for example? I'd suggest proposing a research project to find out why people want to build with The deliverables from such a project could be a requirement spec for "build everything from source" activities and in particular a listing of the constraints that need to be satisfied for any solution. Once we have that documentation, follow-up projects to actually design and/or implement a solution will be much easier to specify. Or it may be that the negative reception that's happened so far can be crystallised into specific issues that we (the Python packaging community) aren't willing to support, which will give users much more actionable feedback at least. |
In this comment, @pfmoore suggests that we may need to re-think
--no-binary
, specifically aroundwheel
but more generally around any or all build dependencies. In pypa/pip#7831, we've essentially concluded that--no-binary :all:
is unsupported for bootstrapping issues. As reported in pypa/wheel#344 and and pypa/wheel#332, wheel is essentially blocked from adopting any form of PEP 518, or any other tool that requires pyproject.toml (because presence of a pyproject.toml implies PEP 518) unless wheel can elect to break--no-binary wheel
installs.Furthermore, the
backend-path
solution is likely not to be viable. Even at it's facebackend-path
adds a path to the PYTHONPATH when building, but unless a source checkout ofwheel
has hadsetup.py egg-info
run on it, it won't have the requisite metadata to advertise its distutils functionality.@pfmoore has suggested that it's not the responsibility of the PyPA or its maintainers to solve the issue, but that the downstream packagers should propose a solution. To that end, should
wheel
restore support for pyproject.toml and break--no-binary
workflows, thus breaking the world for downstream packagers and incentivizing them to come up with a solution (that seems inadvisable)? Shouldwheel
(and other use-cases) remain encumbered with this (undocumented) constraint in order to force legacy behavior when building wheel, thus alleviating the incentive to fix the issue (also sub-optimal)?Perhaps solving the "bootstrapping from source" problem would be a good one for the PSF to sponsor.
The text was updated successfully, but these errors were encountered: