nanobind 3 and a new "split" compilation mode

[wjakob]

Hi everyone!

I've just released a development prerelease of nanobind 3.0.0, which introduces a new distribution model called "split mode". I think that this could be quite appealing to some projects that suffer from having to distribute many Python wheels. You can read about it here.

It would be great if you could test your projects with this split mode and let me know if you encounter any issues before I make a full release. This is a discussion thread about anything related to the new release.

I am especially curious to hear from some of the heavy users of nanobind: @zcbenz @awni @hawkinsp @XXXXRT666 @oremanj.
Another thing worth noting is that other build system adapters for Bazel and Meson will need changes to support split mode. (cc @nicholasjng, @WillAyd).

That's all for now.

Thanks,
Wenzel

[hawkinsp]

Split mode is certainly helpful; previously I've rejected using the stable ABI, not wanting to pay performance costs.

A few questions that occurred to me when I read the docs that I think might be worth adding to the docs:

• This must add an indirection to certain function calls. Is there a performance cost to this, and what kinds of operations are affected?
• What happens if you have direct calls to the Python API yourself, outside nanobind? The answer to this is simple enough, but might be worth stating explicitly: your module is simply targeting the stable ABI for direct CPython API calls, so outside of nanobind you have the usual rules and concerns about the stable ABI.
• How is 3.14t handled, since it does not support abi3t? Answer: it isn't, you just need to not use split mode for this version.

I think it's likely I will use this mode for many but perhaps not all of my extension modules.

[hawkinsp]

We're very careful with, e.g., linker scripts so our shared libraries do not import or export any C++ symbols (except libstdc++), so it's probably safe. (It's not strictly UB or not in the C++ sense; the C++ standard has nothing to say on the topic of shared libraries or symbols, so it's hard to really find a document that specifies what is safe or not here...). We almost assume there will be other copies of LLVM in the process.

I'm not sure what to suggest here. Option 2 is possible but annoying to do.

[wjakob]

I thought some more about this today and changed my mind. I am also concerned about binding clashes making the public backend much less useful. Following some benchmarking, it turns out that the overhead is negligible. I just pushed f4a0c5f which makes NB_DOMAIN usable again with split-mode builds. (And it is specified on the user extension side)

[hawkinsp]

Thanks, I'll take a look. That makes me considerably more comfortable with the idea.

Also on:

3.14t isn't handled, split mode will target 3.15t and later via the abi3t stable ABI. For all situations that split mode cannot support, the easy workaround is to build a non-split wheel and distribute that as usual.

Thinking about this one: it might make the life of users simpler if you support split mode even in Python 3.14t. e.g. I'm slightly vague as to whether it's even possible to specify a nanobind-backend dependency for all except one wheel variant (e.g., NOT for 3.14t)? PEP 780 would allow for that but it's a draft and I'm not sure pip even supports it? And since there's no nanobind-backend for Python 3.14t one must skip that dependency or the package will not be installable. And even if you can do that everyone will have an easier day if you don't have to think about it.

[wjakob]

I've just made a -dev2 release of nanobind and nanobind-backend, which make NB_DOMAIN work on extensions again. It ships a few more performance improvements. Note in particular this change, which requires code adaptations:

Faster sequence construction: creating and filling sequences iteratively using code like

nb::list out;
for (size_t i = 0; i < n; ++i)
    out.append(/* value of entry i */);

is expensive in the limited ABI, which also affects the new split mode. The necessary per-iteration Python library calls and underlying internal checks can add up to significant overheads.

nanobind 3 addresses this issue using the new nb::tuple_builder and nb::list_builder classes. They provide a fast path for constructing tuples and lists of a known size. Use them as follows:

nb::list_builder builder(n);
for (size_t i = 0; i < n; ++i)
    builder.put(/* value of entry i */);
nb::list result = builder.commit();

All internal sequence casters now use this form. Building sequences becomes up to ~1.5x faster when using this approach in split-mode extensions.

[wjakob]

@hawkinsp : I've released a new -dev3 version of nanobind and the backend module containing a few more improvements. I now am happy with this state and plan to release it in roughly a week unless there are problems with it. Could I ask you to give this a try to build your projects? Thanks!!

[nicholasjng]

Nice work!

I'm a bit behind on the Bazel distribution side, although I have not heard of any remaining users of nanobind_bazel, either. The standalone @nanobind BCR module is now maintained by @jwnimmer-tri, so I guess this will be interesting for him as well.

Other than that, I think the Bazel user base is largely going the module extension route, which makes sense since nanobind is a very lightweight dependency. Maybe it's worth documenting that setup instead of publishing opinionated Bazel builds to the BCR that don't fit anyone's needs 100%. I'm curious about other opinions.

[hawkinsp]

Other than that, I think the Bazel user base is largely going the module extension route

Yeah for example we use nanobind via our own module extension at the moment. We did that for a very long time and never switched over. Also sometimes we want to pin unreleased nanobind versions to get a bug fix.

[nicholasjng]

Should nanobind advertise a different Bazel setup path in its RTD documentation?

Yes, the module extension specifically in my opinion. nanobind_bazel has used that internally from the start, and bzlmod has been the de-facto setup in Bazel projects for a while.

I can send a patch if you are interested, this would also retire the nanobind_bazel API docs in favor of a more broadly applicable setup recipe.

[wjakob]

I can send a patch if you are interested, this would also retire the nanobind_bazel API docs in favor of a more broadly applicable setup recipe.

That sounds great!

[jwnimmer-tri]

Thanks for the ping @nicholasjng! I was able to port my Bazel project to non-split v3 this morning (RobotLocomotion/drake#24898), and next up I'll try turning on split mode. I'll work on preparing Bazel Central Registry support for split mode but I might not be able to finish and publish that until after my holiday.

[wjakob]

FYI: I've released a new -dev3 version of nanobind and the backend module.

[hawkinsp]

Another thing I noticed: abi3t support requires split mode. That is unfortunate if one cannot use split mode for the reasons above...

[wjakob]

There is nothing that can be done about that. abi3t did not expose PyVarObject, which makes it unusable to nanobind (which uses this heavily). The split mode was born out of the frustration of not being able to support abi3t :-).

[XXXXRT666]

One issue I noticed while testing nanobind 3 split mode with MLX is its interaction with the CPython buffer protocol.

Nanobind’s split-mode frontend targets the Python 3.10 Stable ABI and defines Py_LIMITED_API=0x030A0000. However, Py_buffer only became part of the Stable ABI in Python 3.11.

MLX implements the buffer protocol for mx.array by registering custom Py_bf_getbuffer and Py_bf_releasebuffer slots. Its getbuffer callback directly fills a Py_buffer, including the data pointer, shape, strides, format, and lifetime information. Consequently, the MLX frontend does not compile against the Python 3.10 Limited API because Py_buffer, PyBUF_FORMAT, and PyBUF_STRIDES are unavailable.

This appears to prevent MLX from using the stock split-mode configuration to publish a single abi3 frontend wheel per platform and architecture across all supported Python versions while preserving memoryview(mx.array) and the existing buffer-protocol behavior.

Is there a recommended solution for custom buffer-protocol exporters in split mode? For example, could split mode support a configurable Stable ABI floor such as cp311-abi3, or provide a backend-side interface for custom buffer exporters so that access to Py_buffer occurs in the CPython-specific backend?

Nanobind’s own nb::ndarray buffer exporter appears to live in the backend, but MLX exposes the buffer protocol on its existing mx.array type, so replacing it with nb::ndarray would not be a drop-in solution.

Is there another supported approach that I may be missing?

[wjakob]

Similar to how the Py_bf_* constants are defined, you could copy-paste the struct Py_buffer contents here. A nanobind backend slot would only help if you needed access to some Py_* style function entry point that is not available.

[XXXXRT666]

It's only a property of your build system (what Py_LIMITED_API you define during build time), though. I guess I do not use any of the build stuff nanobind provides so that's why it was easy to do.

Yeah, nanobind pins Py_LIMITED_API=0x030A0000 in cmake/nanobind-config.cmake

[wjakob]

Targeting the "lower" stable ABIs can actually be a good thing. Some key operations (Py_SIZE, Py_INCREF, Py_DECREF, Py_TYPE, ...) are macros in this stable ABI version, which means that the code can be inlined into the "frontend" part of the extension instead of generating CPython library calls.

Later on, they were converted into opaque library calls.

[XXXXRT666]

Similar to how the Py_bf_* constants are defined, you could copy-paste the struct Py_buffer contents here. A nanobind backend slot would only help if you needed access to some Py_* style function entry point that is not available.

MLX works well with that, all problems get resolved now

[wjakob]

Awesome, great to hear!

[hpkfft]

I haven't had time to look at this "split" mode yet, but one random thought occurred to me: version 2.2.0 (October 3, 2024) mentioned

The old interface (nanobind::ndarray_traits) still exists but is deprecated and will be removed in the next major release.

So, can we remove it? [I just like removing code. 😉]

[wjakob]

Perfect, thanks for bringing this up.

[XXXXRT666]

One additional packaging issue came up while testing MLX on Python 3.13t and 3.14t.

Regular wheels use split mode, and MLX generates stubs during the wheel build by importing mlx.core, so nanobind-backend must already be available in the isolated build environment. Python 3.13t/3.14t cannot use split mode and fall back to linked mode, so those builds must not request nanobind-backend.

This cannot currently be expressed in regular requires before building, cause environment markers do not expose whether python is free-threaded, 3.13 and 3.13t have the same python_version. install_requires is also too late for the stub-generation import.

My current workaround is a wrapper around setuptools.build_meta. It overrides get_requires_for_build_wheel() and get_requires_for_build_editable(), checks sysconfig.get_config_var("Py_GIL_DISABLED"), and adds nanobind-backend only for GIL enabled builds. This is a python issue rather than a nanobind runtime issue.

_build_backend/backend.py

import os
import sysconfig

from setuptools import build_meta as _setuptools_build_meta
from setuptools.build_meta import *  # type: ignore # noqa: F403


def _with_nanobind_backend(requires):
    build_backend = int(os.environ.get("MLX_BUILD_BACKEND_PACKAGE", 0))
    free_threaded = bool(sysconfig.get_config_var("Py_GIL_DISABLED"))
    if not build_backend and not free_threaded:
        requires.append("nanobind-backend>=1.0.0.dev2")
    return requires


def get_requires_for_build_wheel(config_settings=None):
    requires = _setuptools_build_meta.get_requires_for_build_wheel(config_settings)
    return _with_nanobind_backend(requires)


def get_requires_for_build_editable(config_settings=None):
    requires = _setuptools_build_meta.get_requires_for_build_editable(config_settings)
    return _with_nanobind_backend(requires)

pyproject.toml

[build-system]
requires = [
  "setuptools>=80",
  "cmake>=3.26",
  "typing_extensions",
]
build-backend = "backend"
backend-path = ["_build_backend"]

[XXXXRT666]

Not a big problem, just leaving this here in case others run into the same packaging issue