Supporting CPython's Limited API

[amcn]

Starting in CPython 3.2, there is a means for extension writers to target a limited subset of its API which allows an extension module to be compiled once and work with multiple Python versions. The API version is selected by defining a symbol (Py_LIMITED_API) to a given value.

Currently extensions compiled via the Meson python module's extension_module method always target the version of Python that is detected by Meson.

It would be useful to me (and hopefully others too) to be able to target the limited API via extension_module. For example:

python3 = import('python').find_installation('python3')
my_ext_module = python3.extension_module('my_ext_module',
  sources: my_ext_module_sources,
  include_directories: my_ext_include_dirs
  limited_api: '3.5'
)

Is this something others would either be interested in or would have reservations about? I'm opening this discussion to invite comment on this before I start working on adding this.

Some prior art: CMake (recently) added support for this.

[QuLogic]

extension_module is just a wrapper around shared_module and accepts all the same arguments it does. So you can already do something like c_args: ['-DPy_LIMITED_API=0x03050000']

[eli-schwartz]

It's considerably more interesting for built-in support, to handle the name_suffix for you.

There's also a nuance regarding which DLL to link to on Windows; this must be handled by meson itself, there's no other choice.

[eli-schwartz]

This has been previously discussed, but I forget where.

Yes, this would be a useful enhancement, if someone wants to add it. Not particularly high priority though, since e.g. for my own use I'd rather compile locally. :P

It should probably support a module option? since there are performance optimizations that are disabled in the limited api, and people building for reasons other than PyPI distribution may often want to build a more specific but slightly faster binary.

[amcn]

Thanks for your feedback. So I have started working on this tonight and would like to bounce some questions of people:

1. I was planning on keeping the argument format as '3.x' but I am wondering if it would also make sense to support putting in the version hex as well? This would facilitate users who currently set a compiler definition (a la '-DPy_LIMITED_API=0x..') as they could just move the version hex into the meson file and remove the definition. Does this sound like it's worth supporting?

2. As a follow up to the last question: for cases such as the above, where a user had previously been manually specifying '-DPy_LIMITED_API=0x..', what should Meson do? I am wondering if it makes sense to raise a warning pointing them to the new kwarg instead.

[amcn]

2. As a follow up to the last question: for cases such as the above, where a user had previously been manually specifying '-DPy_LIMITED_API=0x..', what should Meson do? I am wondering if it makes sense to raise a warning pointing them to the new kwarg instead.

So I've done a search on GitHub and, unless my search-fu is lacking, there only seems to be one public project that does this, but not using extension_module just using shared_module directly. Huh, I assumed this would be more common.

[eli-schwartz]

What would be the general motivation for end users to want to put a version hex in there instead of a python version? Supporting only one case makes the meson codebase a bit simpler, so if there's no actual demand for using a version hex then I wouldn't bother with it.

The advantage of accepting version numbers and internally converting to a hex is that it's much more readable for a project to use that, so I'd think they will actually be happy at the opportunity to get away from using version hexes.

[amcn]

Huh, so I've discovered that Cython has a similar thing: CYTHON_LIMITED_API which works similarly.

Here's how I plan to deal with that: if a user has supplied a limited_api kwarg and there are Cython files in the input sources, then I add the flag CYTHON_LIMITED_API to cython_args with the same value as the flag Py_LIMITED_API in 'c_args'.

Does this sound like it will work? I'm not sure how to reliably check if the input sources contain any Cython files. I was intending to check the file extension but I'm not sure if that catches all cases.

[amcn]

Implementation posted for review #11745