Question: Simple example

[peekxc]

This is more of a question born out of my own ignorance on Python's build internals. I would like to use extensionlib to build native extension modules interfaced with pybind11. There is a simple example of how to do this with setuptools + setup.py (and other build tools). I'm trying to figure out how to do this with extensionlib + hatch but I can't figure it out.

1. Do I use the custom build hook to compile native extensions, or do I need to make my own build-hook plugin?

Something like:

from hatchling.builders.hooks.plugin.interface import BuildHookInterface
class CustomHook(BuildHookInterface):
    def initialize(self, version, build_data):
        if self.target_name == 'wheel':
            ...

2. ...or does extensionlib scaffold these build hooks via entry-points? If I have something like this:

# pyproject.toml
...
[project.entry-points.extensions]
hatch_build = "my-pkg:ExampleExtensionModules"

then this should instantiate a ExampleExtensionModules class in a hatch_build.py file, yes? And then build things with the build runner? Something like:

# hatch_build.py
from extension.runner import BuildRunner
from extension.interface import ExtensionModules

class ExampleExtensionModules(ExtensionModules):
    def __init__(self, name: str, root: str, metadata: dict, config: dict):
      super().__init__(name, root, metadata, config)
      runner = BuildRunner('.', ...)
      ....

    def inputs():
      return(["src/my_pkg/ext/"]) # this stores my __init__.py + extension module source files 

    ....

I can't seem to parse how to connect the build script with the pyproject.toml file / with hatch build; hatch build --ext silently returns without error and without building anything in the simple example I'm trying to get going now.

[ofek]

Hey! The idea is every extension builder like pybind11 will implement this and then Hatchling will add an always-on build hook that uses https://ofek.dev/extensionlib/runners/

I haven't done that yet because I'm waiting on @henryiii to do a PoC with I think scikit-build?

[peekxc]

I haven't done that yet because I'm waiting on @henryiii to do a PoC with I think scikit-build?

Well although a PoC w/ scikit-build would be informative, I would hope the pybind11 extension runner would be independent of scikit-build. Since I use Meson instead of CMake for building C++, I have no interest in scikit-build.

The idea is every extension builder like pybind11 will implement this and then Hatchling will add an always-on build hook that uses https://ofek.dev/extensionlib/runners/

Ahh, ok this may be outside of my area of expertise. I may poke around at trying this myself but will be watching the pybind11 repo otherwise.

[henryiii]

Pybind11 is a binding tool, not a build system. Scikit-build and mesonpy are build systems, and they would support extensions. I can't speak for mesonpy, but I'll be making sure pybind11 and scikit-build work well together (in fact, as just announced at SciPy, I'm funded to work on scikit-build over next three years). We've also discussed making a "simple" builder that would generate the configuration to build without hand-written CMake code - that's a bit down the road, though.

[pfmoore]

So as an outsider here whose interest is mainly "how does this fit in the packaging ecosystem?" I'm still confused over where this fits.

It sounds like @ofek is saying that this is a protocol that allows PEP 317 build backends (like hatchling), on being told that "file X is an extension managed by extension builder Y", to call Y in a standardised way to build¹ the file X that the backend needs. Which is cool, but what is Y? It sounds like it's something like mesonpy or scikit-build - but I thought those two were PEP 517 backends? So is this for one backend to call another? Or is the expectation that we'll see a crop of a new sort of component, the "extension builder", which focuses on implementing a build process and exposing that functionality via extensionlib?

If enabling a new sort of "extension builder" component is the intention, then this sounds really cool and I look forward to seeing it bear fruit!

Footnotes

1. This actually sounds general enough that why limit it to extensions? I could see other uses, such as a FFI tool that takes a description of a C interface and generates a Python file that wraps that interface using ctypes, with no extension involved. ↩

[ofek]

Yes that is the intention 😄

I could see other uses, such as a FFI tool that takes a description of a C interface and generates a Python file that wraps that interface using ctypes, with no extension involved.

Yes, that would be valid.

This actually sounds general enough that why limit it to extensions?

I think generalizing build hooks should be a separate endeavor.

[henryiii]

mesonpy or scikit-build - but I thought those two were PEP 517 backends

Mesonpy predates this. Scikit-build is gaining a PEP 517 backend because writing another PEP 517 backend is a right of passage for anyone in packaging because the proposal predates this idea & it's a simple "easy start" point for users. Though depending on how this goes, maybe we can just rely on this. I'm not against the idea.

I'm giving a slide today at SciPy where I give an imaginary example:

[build-system]
requires = ["hatchling", "scikit-build-core", "mypyc"]
build-backend = "hatchling.build"

[project]
name = "example"
version = "0.1.0"

[[extensions]]
build-backend = "scikit_build_core.extension"
src = "src1/CMakeLists.txt"

[[extensions]]
build-backend = "scikit_build_core.extension"
src = "src2/CMakeLists.txt"

[[extensions]]
build-backend = "mypyc.extension"
src = "src/*.py"

I was imagining it would look something like that.

[ofek]

As currently implemented it would look like:

[build-system]
requires = ["hatchling", "scikit-build-core", "mypyc"]
build-backend = "hatchling.build"

[project]
name = "example"
version = "0.1.0"

[[project.extensions.scikit-build]]
src = "src1/CMakeLists.txt"

[[project.extensions.scikit-build]]
src = "src2/CMakeLists.txt"

[[project.extensions.mypyc]]
src = "src/*.py"

[henryiii]

Extensions are not logically part of the project metadata. If they were under anything, they'd be under [[build-system.extensions]].

IMO, the extension build backend ideally should have the same structure as selecting the build backend; it's a familiar interface, and allows a package to provide more than one extension build backend. We'll also need to think about how we'd send different arbitrary configuration to each extension, which might bias is to a .<package-name>.

(Totally fine to have the above API for a draft to work on, though, just sharing my opinions).

[ofek]

1. [[build-system.extensions]] is much better!
2. A package can define an arbitrary number of extension builders https://ofek.dev/extensionlib/builders/#plugin-registration

[pfmoore]

PEP 518 disallows anything outside of [tool], so [[build-system.extensions]] (or [[project.extensions]]) would need to be standardised in a PEP.

[henryiii]

Or even [[extensions]], yes. I was thinking the PoC (here) would use [[tool.extensionlib.extensions]] as the placeholder. I’m also hoping to see if we can propose a Protocol vs. requiring a library / ABC (though extensionlib would provide the Protocol and helpers).

[gaborbernat]

PEP 518 disallows anything outside of [tool], so [[build-system.extensions]] (or [[project.extensions]]) would need to be standardised in a PEP.

You've stated your preference recently that any PEP should have an initial implementation and use before it's standardised, to avoid adopting PEPs that then are not used then. This seems to contradict then PEP-518, because I'd rather not need to migrate to a whole new namespace once it's approved 🤔

[pfmoore]

I’m also hoping to see if we can propose a Protocol vs. requiring a library / ABC

If I'm understanding correctly, a standard would be essentially like PEP 517, defining a function that modules implementing the PEP should provide. I guess that's basically a protocol (albeit simply in words, rather than in the sense of typing.Protocol)?

If you are going to standardise this, though, what you propose in the PEP is up to you (and whoever you get as PEP sponsor). Feel free to ignore my comments if they aren't helpful.

This seems to contradict then PEP-518, because I'd rather not need to migrate to a whole new namespace once it's approved

Agreed, that's unfortunate, but necessary if you want to be pedantic.

I don't recall the statement you're referring to, so I'm reluctant to try to clarify "what I meant" because I fear I'll get called out if I'm inconsistent. But I will say that my main concern is getting something that we have a reasonable assurance that any PEP I accept isn't going to fall apart at the seams when people try to implement it. If someone can do that without a prototype implementation, then that's fine. And if someone else volunteers to be PEP-delegate, they can pick their own criteria.

But can we park the PEP process debates for now? I don't want to hijack this issue with off-topic discussions.

[peekxc]

I guess my core confusion was I thought extensionlib (+hatchling) was at its core providing an alternative to distutils + setup.py--essentially doing what scikit-build-core does (or is aspiring to do in the future when disutils is removed as a dependency?), but @pfmoore cleared things up.

I could see other uses, such as a FFI tool that takes a description of a C interface and generates a Python file that wraps that interface using ctypes, with no extension involved.

This sounds to me like a fairly non-trivial task (perhaps already achieved to some degree by Cython?), but I could be wrong. Pybind11 provides this non-trivial FFI-functionality already for me in C++, which is really what I'm concerned with. I guess ctypes might be the Python standards-compliant way to go about it with C, but that's beyond the scope of my original question.

[eli-schwartz]

Well although a PoC w/ scikit-build would be informative, I would hope the pybind11 extension runner would be independent of scikit-build. Since I use Meson instead of CMake for building C++, I have no interest in scikit-build.

If you're already prepared to use Meson, then Meson implements its own classification for:

• installing source files (pymod.install_sources(....))
• building and installing compiled extensions (pymod.extension_module(...))

In the latter case, you'd be building it much the same way you'd build a C/C++ library -- although it's actually a subclass of C/C++ libraries that builds one with the .cpython-310-x86_64-linux-gnu.so style extension -- simply define the library name, provide source files and cpp_args and link_args, and any dependencies it needs to be linked to. pybind11 does not need anything coupled to an extension runner (?) for this, because it provides dependency lookup methods (pybind11-config, a cmake package, a Meson wrap) and it is really as simple as adding the headers path to the extension you want to build. So you can simply do dependency('pybind11') in Meson and carry on.

I'm not sure there's any great need in that case for a protocol to tell Meson how to either build extensions or add dependencies to them. mesonpy, the PEP 517 build backend, might choose to do so... I guess the idea would be to run one meson project per extension? The cmake examples seem to imply that will be the case. But currently, mesonpy has a satisfactory solution using one meson project per wheel, and leveraging Meson's (python-aware) output manifest.

...

As far as an example of using extensionlib goes, from my limited perusal it seems like a hook-based approach to code your own simple build system for performing tasks, including but not limited to C extensions. It's probably quite useful for a PEP 517 backend that doesn't have that built in, but it's not clear to me how advantageous that is for stepping out of a general-purpose build system such as Meson or CMake.

Am I correct in assuming that this is really just a replacement for hatch_build.py?