Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support stable Python ABI #1613

Closed
wants to merge 3 commits into from
Closed

Support stable Python ABI #1613

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
363f8d9
Support stable Python ABI.
vadimcn Aug 13, 2019
d888e4a
Check Py_LIMITED_API value, if predefined.
vadimcn Dec 22, 2019
1a908af
Document linkage requirements.
vadimcn Dec 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 10 additions & 0 deletions .travis.yml
View file
Open in desktop
Expand Up @@ -245,6 +245,16 @@ matrix:
env: SWIGLANG=python SWIG_FEATURES=-builtin PY3=3 VER=3.7 SWIGOPTPY3=
sudo: required
dist: xenial
- compiler: gcc
os: linux
env: SWIGLANG=python SWIG_FEATURES=-py3-stable-abi PY3=3
sudo: required
dist: xenial
- compiler: gcc
os: linux
env: SWIGLANG=python SWIG_FEATURES="-py3-stable-abi -O" PY3=3
sudo: required
dist: xenial
- compiler: gcc
os: linux
env: SWIGLANG=python SWIG_FEATURES=-O
Expand Down
71 changes: 71 additions & 0 deletions Doc/Manual/Python.html
View file
Open in desktop
Expand Up @@ -138,6 +138,7 @@ <H1><a name="Python">32 SWIG and Python</a></H1>
<li><a href="#Python_nn76">Abstract base classes</a>
<li><a href="#Python_nn77">Byte string output conversion</a>
<li><a href="#Python_2_unicode">Python 2 Unicode</a>
<li><a href="#Python_3_stable_abi">Stable ABI</a>
</ul>
<li><a href="#Python_multithreaded">Support for Multithreaded Applications</a>
<ul>
Expand Down Expand Up @@ -964,6 +965,7 @@ <H3><a name="Python_commandline">32.2.9 Additional Python commandline options</a
<tr><td>-nothreads </td><td>Disable thread support for the entire interface</td></tr>
<tr><td>-olddefs </td><td>Keep the old method definitions when using -fastproxy</td></tr>
<tr><td>-py3 </td><td>Generate code with Python 3 specific features and syntax</td></tr>
<tr><td>-py3-stable-abi </td><td>Generate code compatible with Python 3 stable ABI (PEP 384)</td></tr>
<tr><td>-relativeimport </td><td>Use relative Python imports</td></tr>
<tr><td>-threads </td><td>Add thread support for all the interface</td></tr>
<tr><td>-O </td><td>Enable the following optimization options: -fastdispatch -fastproxy -fvirtual</td></tr>
Expand Down Expand Up @@ -7228,6 +7230,75 @@ <H3><a name="Python_2_unicode">32.12.5 Python 2 Unicode</a></H3>
prohibiting it.
</p>

<H3><a name="Python_3_stable_abi">32.12.5 Python 3 Stable ABI</a></H3>

<p>
Quoting from <a href="https://docs.python.org/3/c-api/stable.html">Python 3 documentation</a>:
<blockquote>

<p>
Traditionally, the C API of Python will change with every release.
Most changes will be source-compatible, typically by only adding API, rather
than changing existing API or removing API (although some interfaces do get
removed after being deprecated first).
</p>

<p>
Unfortunately, the API compatibility does not extend to binary compatibility (the ABI).
The reason is primarily the evolution of struct definitions, where addition of a new field,
or changing the type of a field, might not break the API, but can break the ABI.
As a consequence, extension modules need to be recompiled for every Python release
(although an exception is possible on Unix when none of the affected interfaces are used).
In addition, on Windows, extension modules link with a specific pythonXY.dll and need to be
recompiled to link with a newer one.
</p>

<p>
Since Python 3.2, a subset of the API has been declared to guarantee a stable ABI.
Extension modules wishing to use this API (called “limited API”) need to define Py_LIMITED_API.
A number of interpreter details then become hidden from the extension module; in return,
a module is built that works on any 3.x version (x>=2) without recompilation.
</p>

<p>
In some cases, the stable ABI needs to be extended with new functions. Extension modules
wishing to use these new APIs need to set Py_LIMITED_API to the PY_VERSION_HEX value
(see API and ABI Versioning) of the minimum Python version they want to support
(e.g. 0x03030000 for Python 3.3). Such modules will work on all subsequent Python
releases, but fail to load (because of missing symbols) on the older releases.
</p>

<p>
The set of functions available to the limited API is documented in <a href="https://www.python.org/dev/peps/pep-0384/">PEP 384</a>.
</p>

</blockquote>
</p>

<p>
The <tt>-py3-stable-abi</tt> option enables compatibility of code generated by SWIG with stable ABI. SWIG modules that use stable ABI will be able to work
with Python versions 3.4 and above without recompilation.

<p> However, this compatibility comes at a performance cost: some Python API macros are not available in this mode,
so SWIG must use function versions of the same. For example, instead of <tt>PyTuple_GET_ITEM</tt> SWIG has to use <tt>PyTuple_GetItem</tt>.
Unlike the macro, <tt>PyTuple_GetItem</tt> cannot be inlined, so it will execute slower. Also, please note that <tt>-py3-stable-abi</tt> is not compatible
with some of the other SWIG Python options such as <tt>-builtin</tt> or <tt>-fastproxy</tt>.
</p>

<H4>Linkage</H4>
<p>
Version-independent modules have special requirements when it comes to linking: they should not depend on the versioned Python dynamic library, otherwise the dynamic
linker will refuse to load them into a different version of Python runtime.
Here's <a href="https://www.python.org/dev/peps/pep-0384/#linkage">PEP 384 guidance on linkage</a>. In practical terms, it mean this:
<ul>
<li>On Windows, link to the version-independent <tt>python3.lib</tt>.</li>
<li>On Linux, don't link to <tt>libpython</tt> at all, leaving all imported Python symbols unresolved. Make sure that unresolved symbols are not disallowed via "<tt>-z defs</tt>" linker option.</li>
<li>On MacOS, also don't link to <tt>libpython</tt>, but you will need to explicitly allow unresolved symbols via "<tt>-undefined dynamic_lookup</tt>" option.
If you're not invoking linker directly, you may need to instruct your compiler to do so. For example, GCC-compatible complers will need this flag:
"<tt>-Wl,-undefined,dynamic_lookup</tt>".</li>
</ul>
</p>

<H2><a name="Python_multithreaded">32.13 Support for Multithreaded Applications</a></H2>


Expand Down
6 changes: 5 additions & 1 deletion Examples/test-suite/python/Makefile.in
View file
Open in desktop
Expand Up @@ -67,7 +67,6 @@ CPP_TEST_CASES += \
python_moduleimport \
python_overload_simple_cast \
python_pickle \
python_pybuffer \
python_pythoncode \
python_richcompare \
python_strict_unicode \
Expand All @@ -77,6 +76,11 @@ CPP_TEST_CASES += \
swigobject \
template_matrix \

# buffers are not supported with -py3-stable-abi
ifeq (,$(findstring -py3-stable-abi,$(SWIG_FEATURES)))
CPP_TEST_CASES += python_pybuffer
endif

# li_std_carray
# director_profile

Expand Down
11 changes: 11 additions & 0 deletions Lib/python/pyhead.swg
View file
Open in desktop
Expand Up @@ -85,3 +85,14 @@ SWIG_Python_str_FromChar(const char *c)
#define PyDescr_NAME(x) (((PyDescrObject *)(x))->d_name)
#define Py_hash_t long
#endif

#ifdef Py_LIMITED_API
# define PyTuple_GET_ITEM PyTuple_GetItem
# define PyTuple_GET_SIZE PyTuple_Size
# define PyCFunction_GET_FLAGS PyCFunction_GetFlags
# define PyCFunction_GET_FUNCTION PyCFunction_GetFunction
# define PyCFunction_GET_SELF PyCFunction_GetSelf
# define PyList_GET_ITEM PyList_GetItem
# define PyList_SET_ITEM PyList_SetItem
# define PySliceObject PyObject
#endif
25 changes: 24 additions & 1 deletion Lib/python/pyinit.swg
View file
Open in desktop
Expand Up @@ -15,9 +15,11 @@ extern "C" {

/* Method creation and docstring support functions */

#ifndef Py_LIMITED_API
SWIGINTERN PyMethodDef *SWIG_PythonGetProxyDoc(const char *name);
SWIGINTERN PyObject *SWIG_PyInstanceMethod_New(PyObject *SWIGUNUSEDPARM(self), PyObject *func);
SWIGINTERN PyObject *SWIG_PyStaticMethod_New(PyObject *SWIGUNUSEDPARM(self), PyObject *func);
#endif // Py_LIMITED_API

#ifdef __cplusplus
}
Expand Down Expand Up @@ -146,6 +148,7 @@ swig_varlink_setattr(swig_varlinkobject *v, char *n, PyObject *p) {
SWIGINTERN PyTypeObject*
swig_varlink_type(void) {
static char varlink__doc__[] = "Swig var link object";
#ifndef Py_LIMITED_API
static PyTypeObject varlink_type;
static int type_init = 0;
if (!type_init) {
Expand Down Expand Up @@ -200,12 +203,29 @@ swig_varlink_type(void) {
return NULL;
}
return &varlink_type;
#else
PyType_Slot slots[] = {
{ Py_tp_dealloc, (void*)swig_varlink_dealloc },
{ Py_tp_getattr, (void*)swig_varlink_getattr },
{ Py_tp_setattr, (void*)swig_varlink_setattr },
{ Py_tp_repr, (void*)swig_varlink_repr },
{ Py_tp_str, (void*)swig_varlink_str },
{ Py_tp_doc, (void*)varlink__doc__ },
{ 0, NULL }
};
PyType_Spec spec = {};
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These empty initializer lists pose a problem for some compilers, notably more recent versions of MSVC, or when using strict compile options (e.g. gcc with -Wpedantic).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is indeed not ISO C and we should avoid it:

$ echo 'struct { int x; } a = {};'|gcc -pedantic -x c - -o /dev/null
<stdin>:1:23: warning: ISO C forbids empty initializer braces [-Wpedantic]

The reason for this is presumably to 0-initialise members not explicitly initialised below. The Python docs for this structure explicitly enumerate the members, and the only one not explicitly initialised here is itemsize, so we should be able to drop the {} and add spec.itemsize = 0; below.

There seem to be two more instances of this.

spec.name = "swigvarlink";
spec.basicsize = sizeof(swig_varlinkobject);
spec.flags = Py_TPFLAGS_DEFAULT;
spec.slots = slots;
return (PyTypeObject*)PyType_FromSpec(&spec);
#endif
}

/* Create a variable linking object for use later */
SWIGINTERN PyObject *
SWIG_Python_newvarlink(void) {
swig_varlinkobject *result = PyObject_NEW(swig_varlinkobject, swig_varlink_type());
swig_varlinkobject *result = PyObject_New(swig_varlinkobject, swig_varlink_type());
if (result) {
result->vars = 0;
}
Expand Down Expand Up @@ -318,6 +338,7 @@ SWIG_Python_FixMethods(PyMethodDef *methods,
* Method creation and docstring support functions
* ----------------------------------------------------------------------------- */

#ifndef Py_LIMITED_API
/* -----------------------------------------------------------------------------
* Function to find the method definition with the correct docstring for the
* proxy module as opposed to the low-level API
Expand Down Expand Up @@ -372,6 +393,8 @@ SWIGINTERN PyObject *SWIG_PyStaticMethod_New(PyObject *SWIGUNUSEDPARM(self), PyO
return PyStaticMethod_New(func);
}

#endif // Py_LIMITED_API

#ifdef __cplusplus
}
#endif
Expand Down