Implement as PEP 517 build backend interface, configured with pyproject.toml and CMakeLists.txt

[thewtex]

With the end goal of being a build system tool for PEP 518 with good C extension support, scikit-build should be implemented as a PEP 516 build tool providing the build backend interface specified in PEP 517.

Following the vision specified at the end of this SciPy talk, most of the packaging specification should be provided declaratively in a pypackage.toml pyproject.toml in the [tool.skbuild] section. Additional, more complicated packaging information, like how to build the C extensions, is provided in the CMake CMakeLists.txt configuration.

This means, like flit, we generate wheels directly by creating the wheel layout specified in PEP 427. These wheels can be generated by calling an skbuild script followed by the subcommands specified in PEP 516 or by the API specified in PEP 517. We will create a shim (that can also be placed in setup.py ) when python setup.py install is called that will install the generated wheel so skbuild functionality can be used immediately with pip, a manual call to python setup.py install, or conda. But, if we follow PEP 516, 517, 518, when these PEPs are finalized and implemented in pip and conda, the setup.py shim will no longer be needed.

Steps forward:

• 1. Implement skbuild metadata subcommand. This is the metadata subcommand specified in PEP 516. It outputs the METADATA file contents (see PEP 427) to stdout. These are the metadata keys described in PEP 345. A package that uses scikit-build should specify these in a pypackage.toml file in the [tool.skbuild] section, probably with lower-case key variants, a la flit. We can use wheel.pkginfo to help with this and pytoml
• 2. Implement skbuild.api.get_wheel_metadata. This implements the get_wheel_metadata method as specified in PEP-517, which creates the wheel {package}-{version}.distinfo directory in the specified location and populates it with the METADATA file from Step 1. along with the WHEEL file (see PEP 427).
• 3. Implement the skbuild wheel subcommand. This is the PEP 516 wheel subcommand. Internally, it should call skbuild.api.build_wheel as specified in PEP 517. This can be created with the assistance of wheel.archive. Our [tool.skbuild] section of pypackage.toml can have additional keys, py_modules, a list of Pure Python modules to install directly into the wheel, py_packages, a list of Pure Python packages to install directly into the wheel, and data, directories to install directly into the .data/data section of the wheel. Also, a scripts with a list of script, entrypoint tuples. Then cmaker should be configured with CMAKE_INSTALL_PREFIX set to the wheel root directory. Other CMake variables to specify during configuration: WHEEL_PURELIB, which is set to {package}-{version}.data/purelib, and similar for WHEEL_PLATLIB, WHEEL_HEADERS, and WHEEL_DATA. Finally, there should a section in pypackage.toml that specifies what CMake install command COMPONENTS should be installed.
• 4. Implement the skbuild install subcommand. This can just delegate to wheel.install as a convenient way to install a package. Check out flit install.
• 5. Implement the setup.py shim. This can effectively call skbuild install when the install command is passed to setup.py and skbuild wheel when the bdist_wheel argument is passed to setup.py .
• 6. Implement the skbuild build_requires subcommand. This is the PEP 516 build_requires subcommand, which should call skbuild.api.get_build_requires as specified in PEP 517. This could output the content of a pypackage.toml metadata entry along with other requirements like a cmake python package that encapsulates CMake, etc.
• 7. Implement the skbuild develop subcommand. This is the PEP 516 develop subcommand. Just error out; I don't think it is reasonable to support this.
• 8. Consider a skbuild sdist subcommand. Investigate creating an sdist by putting the contents of git archive along with a PKG-INFO file with the METADATA contents in {package}-{version}.tar.gz output.

The current approach of wrapping setuptools does not get us closer to PEP 516/517/518 support. The monkeypatching of setuptools, which is monkeypatching distutils, and trying to masquerade C-extension binaries as package_data is problematic (see e.g. #117, #106, #84, #107). It does not move towards better pypackage.toml adoption and independence from distutils/setuptools. It should be abandoned.

[jcfr]

@thewtex If I didn't mention it already, outstanding description 💯

[stefansjs]

This looks great. 👍

Can I contribute to its implementation? Some introductory info on where to dive into code would be great 😁

[thewtex]

@stefansjs contributions welcome. The best path forward is to start knocking out the proposed steps, adding unit tests along the way. The current infrastructure is setuptool's based, so this will be mostly new ground to tread. But, factoring out pieces of the current codebase that can be re-used may make sense in some cases.

[dalcinl]

@thewtex @stefansjs Given that scikit-build depends on setuptools, looks like adding pep-517 support may be relatively easy, as long as you are wiling to let setuptools do the heavy lifting. I have things mostly working in mpi4py branch feature/scikit-build.

I'm setting up a local in-tree backend that is a thin wrapper around setuptools. The code is a bit convoluted, because I set things up such that users has to export MPI4PY_BUILD_BACKEND=skbuild to use scikit-build. But my POC shows that some preliminary support could be easily added.

[joreiff]

Setuptools 61 (released 2022-03-24) allows specifying package metadata (name, homepage, etc.) in pyproject.toml. Although scikit-build extends Setuptools, it does not seem to support this yet. This new Setuptools feature, however, may make the implementation in scikit-build a lot easier.