Add Python Raw memory helper functions #2537
Conversation
Cython/Includes/cpython/mem.pxd
Outdated
| @@ -22,6 +24,15 @@ cdef extern from "Python.h": | |||
| # for the I/O buffer escapes completely the Python memory | |||
| # manager." | |||
|
|
|||
| IF PY_VERSION_HEX >= 0x03040000: | |||
There was a problem hiding this comment.
No, that's not how this works. The Python version only comes into play at C compilation time, not during the C code generation.
Instead, add the fallback code to ModuleSetupCode.c and always define the functions here.
There was a problem hiding this comment.
Ok, thanks. Knew something was off, but wasn't sure how to fix it. Will update.
|
Tried to update it. Though am less familiar with the inner workings of Cython. So please let me know if needs to change somehow. Thanks again for your help. |
|
If you have another chance to look, @scoder, would really appreciate it. Seems like there is some issue with this change (probably me doing something incorrectly) and could use some advice. 😄 |
Cython/Includes/cpython/mem.pxd
Outdated
| @@ -1,3 +1,10 @@ | |||
| from cpython.version cimport PY_VERSION_HEX | |||
|
|
|||
| cdef extern from *: | |||
There was a problem hiding this comment.
Just keep these together with the others below. They are all defined in Python.h.
Cython/Utility/ModuleSetupCode.c
Outdated
| @@ -441,6 +441,12 @@ class __Pyx_FakeReference { | |||
| #define PyObject_Realloc(p) PyMem_Realloc(p) | |||
| #endif | |||
|
|
|||
| #if CYTHON_COMPILING_IN_CPYTHON && PY_VERSION_HEX < 0x03040000 | |||
There was a problem hiding this comment.
No need to restrict this to CPython. PyMem_Malloc() & friends should generally be available. I would add a && !defined(PyMem_Malloc), though, just in case. PyPy uses C-defines for all of the C-API, for example.
Also, more accurately, these functions were first added in 0x030400A1, as you can see from the tags here: python/cpython@0507bf5
There was a problem hiding this comment.
Do the PyPy ones work without the GIL? Don't really know the PyPy C API so had left them out.
There was a problem hiding this comment.
Can't say. But if it's a documented C-API feature of these functions to allow that, then it's simply PyPy's fault if it doesn't support that. I don't see why it should be up to Cython to decide whether to provide these names at all.
|
Sorry, the test failures were my fault. Just pushed a test fix to master. Could you please rebase your branch? |
In Python 3.4+, Raw memory helper functions were added to wrap, track, and check `malloc` and `free` calls for C memory outside of the GIL. These have the same API as the existing `PyMem_*` calls except they use Raw and never require the GIL. For Python 2/3 compatibility, export the existing `PyMem_` functions on Python 2 as `PyMem_Raw` where they act basically equivalently. These are especially important as Python 3.6 changed the existing `PyMem_` functions to allocate from pymalloc instead of the C allocator meaning the GIL must now be held with them. So these are Python 2/3 alternatives that can be relied on to not require the GIL.
To handle the Python version differences, handle the definitions of `PyMem_Raw*` functions in `ModuleSetupCode`. Then extern them in `cpython.mem` without using a version check or a specific header.
jakirkham
force-pushed
the
extern_py_raw_mem
branch
from
c711137
to
8ea8e0f
Compare
Technically these functions were around in some of the Python 3.4 prereleases. So add a more accurate version constraint.
jakirkham
force-pushed
the
extern_py_raw_mem
branch
from
8ea8e0f
to
fb15825
Compare
| @@ -1,3 +1,5 @@ | |||
| from cpython.version cimport PY_VERSION_HEX | |||
There was a problem hiding this comment.
This looks like a left-over now.
|
BTW, what's missing now is a test that shows that these three functions actually work, across all Python versions. |
In Python 3.4+, Raw memory helper functions were added to wrap, track, and check
mallocandfreecalls for C memory outside of the GIL. These have the same API as the existingPyMem_*calls except they use Raw and never require the GIL. For Python 2/3 compatibility, export the existingPyMem_functions on Python 2 asPyMem_Rawwhere they act basically equivalently. These are especially important as Python 3.6 changed the existingPyMem_functions to allocate from pymalloc instead of the C allocator meaning the GIL must now be held with them. So these are Python 2/3 alternatives that can be relied on to not require the GIL.