Idea: Capture build dependencies for complicated wheels in pyproject.toml

[chrisantonellis]

Hello! I'm looking for feedback on the following idea. I'd be happy to make a contribution to this project

When a python package doesn't provide a manylinux wheel on PyPI when it could, what do you do? Build it yourself. Or what if you want the package linked against a different version of a system library? Build it yourself.

This becomes expensive when operating at scale, building and hosting internally compiled python packages for large organizations is a bummer. In part because the "recipes" for building complicated wheels are only captured in project and platform specific documentation. I'm looking at you pandas.

What if the data required to modify a build environment was captured in "steps" similar to the CI configurations used by this tool? Set environment variables, install system_deps-1.0.0, install python_deps==1.0.0, run bash steps etc.

With this info captured as part of pyproject.toml it would be versioned, and could be contributed to by consumers of the package.

[manylinux_build_specification.manylinux2014]
extra_base_system_repositories = ["http://foo.com/packages/"]
system_dependencies = ["foo-1.0.0", "bar-1.0.0"]
python_dependencies = ["foo==1.0.0", "bar==1.0.0"]
environment_variables = ["FOO=BAR"]
steps = [
  "./scripts/build_and_upload.sh --my_option"
]

thanks for any and all input!

Note: This has been shared with the manylinux folks (pypa/manylinux#945), the python ideas mailing list (https://mail.python.org/archives/list/python-ideas@python.org/thread/GJX77GBYLTNOICXFHHHN5JOBKZM43O25/) and the python discussion board (https://discuss.python.org/t/build-dependency-specification-for-manylinux-wheels-idea-for-a-pep/6589)

[Czaki]

The discussion about this already start here: #547
Please move your suggestion there to keep everything in one place.

[YannickJadoul]

@Czaki, this is not fully the same though. This seems to be on a slightly higher level of abstraction?

[henryiii]

Yes (although it might be better as a discussion, it probably should not be the same discussion). Should be cross-linked, at least, though.

[henryiii]

We could pretty easily handle this if it was accepted upstream, I think.

[YannickJadoul]

I was also a bit skeptical toward cibuildwheel providing this, at first. But who else would, upstream?

Also, we do need something that's not "just" manylinux, though. If this is useful/worth it for Linux, then also for other platforms, I'd think.

[chrisantonellis]

crossposted! thanks for the input. I'll start the conversation over there to see if my idea has any merit, then bring it back here if so. thank you