Description
Overview
I recently created the following BPO:
as well as associated PR:
to fix (what I felt was) a bug in (at least) Python 3.8/Python 3.9.
However, I received push-back from at least one commenter on the Python mailing list that the issue was discussing was an "API misuse" of Python.h
(maybe that's true).
Anyway, since this came about because of Cython, I thought it was worth discussing it here.
Actual issue
If you Cythonate something like:
float(1)
then you end-up with some code that looks like this in your output:
static CYTHON_UNUSED double __Pyx__PyBytes_AsDouble(PyObject *obj, const char* start, Py_ssize_t length) {
double value;
Py_ssize_t i, digits;
const char *last = start + length;
char *end;
while (Py_ISSPACE(*start))
start++;
...
And where Py_ISSPACE
has an expansion like:
#define Py_ISSPACE(c) (_Py_ctype_table[Py_CHARMASK(c)] & PY_CTF_SPACE)
(in pyctype.h
)
and where the variable _Py_ctype_table
looks like this:
PyAPI_DATA(const unsigned int) _Py_ctype_table[256];
Problematically, the inclusion of (cpython
) pyctype.h
in Python.h
does not guard _Py_ctype_table
against the case of C++ compilation.
This then leads to the following issue:
- You use pre-compiled version of Python38 (e.g., on Windows), which has been compiled with a C compiler
- This means that all symbols are not name-managled
- You Cythonate some code and pull in a C++-only header
- You then compile your Cythonated code as C++ (e.g., with
/TP
if you're using Visual Studio) - Finally, you try to link your Cythonated object code with
Python38.lib
This then falls apart because Python38.lib has a un-name-mangled version of _Py_ctype_table
, while your Cythonated code has C++-name-managled version of _Py_ctype_table
, which means the linker cannot resolve the symbol, giving a link time error:
cl /Fe:test.exe /TP /I include cython_output.c test.c /link libs/python39.lib
Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29336 for x64
Copyright (C) Microsoft Corporation. All rights reserved.
cython_output.c
test.c
Generating Code...
Microsoft (R) Incremental Linker Version 14.28.29336.0
Copyright (C) Microsoft Corporation. All rights reserved.
/out:test.exe
libs/python39.lib
cython_output.obj
test.obj
Creating library test.lib and object test.exp
cython_output.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) unsigned int const * const _Py_ctype_table" (__imp_?_Py_ctype_table@@3QBIB) referenced in function "double __cdecl __Pyx__PyBytes_AsDouble(struct _object * __ptr64,char const * __ptr64,__int64)" (?__Pyx__PyBytes_AsDouble@@YANPEAU_object@@PEBD_J@Z)
test.exe : fatal error LNK1120: 1 unresolved externals
Resolution
Without changing the use of Py_ISSPACE
within Cython, I feel that the only solution is to change the generated C code to look like this:
#ifdef __cplusplus
extern "C" {
#endif
#include <Python.h>
#ifdef __cplusplus
}
#endif
Personally, given that other files downstream of Python.h
do have extern "C"
, I felt that this really is a Python bug rather than a "user" bug.
However, I'm sharing this in here in case my feelings are off-base and, indeed, this is a Cython bug.
If this is considered a Cython bug, I'm more than happy to make a PR with the above change.