Permalink
Browse files

PyMem_Malloc() now uses the fast pymalloc allocator

Issue #26249: PyMem_Malloc() allocator family now uses the pymalloc allocator
rather than system malloc(). Applications calling PyMem_Malloc() without
holding the GIL can now crash: use PYTHONMALLOC=debug environment variable to
validate the usage of memory allocators in your application.
  • Loading branch information...
vstinner committed Apr 22, 2016
1 parent 5439fc4 commit f5c4b99034fae12ac2b9498dd12b5b3f352b90c8
Showing with 58 additions and 32 deletions.
  1. +37 −23 Doc/c-api/memory.rst
  2. +5 −6 Doc/using/cmdline.rst
  3. +6 −0 Doc/whatsnew/3.6.rst
  4. +7 −0 Misc/NEWS
  5. +3 −3 Objects/obmalloc.c
@@ -165,15 +165,17 @@ The following function sets, modeled after the ANSI C standard, but specifying
behavior when requesting zero bytes, are available for allocating and releasing
memory from the Python heap.
The default memory block allocator uses the following functions:
:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call
``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes.
By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`.
.. warning::
The :term:`GIL <global interpreter lock>` must be held when using these
functions.
.. versionchanged:: 3.6
The default allocator is now pymalloc instead of system :c:func:`malloc`.
.. c:function:: void* PyMem_Malloc(size_t n)
Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
@@ -295,15 +297,32 @@ Customize Memory Allocators
Enum used to identify an allocator domain. Domains:
* :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`,
:c:func:`PyMem_RawRealloc`, :c:func:`PyMem_RawCalloc` and
:c:func:`PyMem_RawFree`
* :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`,
:c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc` and :c:func:`PyMem_Free`
* :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`,
:c:func:`PyObject_Realloc`, :c:func:`PyObject_Calloc` and
:c:func:`PyObject_Free`
.. c:var:: PYMEM_DOMAIN_RAW
Functions:
* :c:func:`PyMem_RawMalloc`
* :c:func:`PyMem_RawRealloc`
* :c:func:`PyMem_RawCalloc`
* :c:func:`PyMem_RawFree`
.. c:var:: PYMEM_DOMAIN_MEM
Functions:
* :c:func:`PyMem_Malloc`,
* :c:func:`PyMem_Realloc`
* :c:func:`PyMem_Calloc`
* :c:func:`PyMem_Free`
.. c:var:: PYMEM_DOMAIN_OBJ
Functions:
* :c:func:`PyObject_Malloc`
* :c:func:`PyObject_Realloc`
* :c:func:`PyObject_Calloc`
* :c:func:`PyObject_Free`
.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)
@@ -328,18 +347,12 @@ Customize Memory Allocators
.. c:function:: void PyMem_SetupDebugHooks(void)
Setup hooks to detect bugs in the following Python memory allocator
functions:
- :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
:c:func:`PyMem_RawCalloc`, :c:func:`PyMem_RawFree`
- :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc`,
:c:func:`PyMem_Free`
- :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
:c:func:`PyObject_Calloc`, :c:func:`PyObject_Free`
Setup hooks to detect bugs in the Python memory allocator functions.
Newly allocated memory is filled with the byte ``0xCB``, freed memory is
filled with the byte ``0xDB``. Additional checks:
filled with the byte ``0xDB``.
Runtime checks:
- Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
allocated by :c:func:`PyMem_Malloc`
@@ -377,8 +390,9 @@ to 512 bytes) with a short lifetime. It uses memory mappings called "arenas"
with a fixed size of 256 KB. It falls back to :c:func:`PyMem_RawMalloc` and
:c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes.
*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_OBJ` domain
(ex: :c:func:`PyObject_Malloc`).
*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_MEM` (ex:
:c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_OBJ` (ex:
:c:func:`PyObject_Malloc`) domains.
The arena allocator uses the following functions:
@@ -628,12 +628,11 @@ conflict.
Set the family of memory allocators used by Python:
* ``malloc``: use the :c:func:`malloc` function of the C library
for all Python memory allocators (ex: :c:func:`PyMem_RawMalloc`,
:c:func:`PyMem_Malloc` and :c:func:`PyObject_Malloc`).
* ``pymalloc``: :c:func:`PyObject_Malloc`, :c:func:`PyObject_Calloc` and
:c:func:`PyObject_Realloc` use the :ref:`pymalloc allocator <pymalloc>`.
Other Python memory allocators (ex: :c:func:`PyMem_RawMalloc` and
:c:func:`PyMem_Malloc`) use :c:func:`malloc`.
for all domains (:c:data:`PYMEM_DOMAIN_RAW`, :c:data:`PYMEM_DOMAIN_MEM`,
:c:data:`PYMEM_DOMAIN_OBJ`).
* ``pymalloc``: use the :ref:`pymalloc allocator <pymalloc>` for
:c:data:`PYMEM_DOMAIN_MEM` and :c:data:`PYMEM_DOMAIN_OBJ` domains and use
the :c:func:`malloc` function for the :c:data:`PYMEM_DOMAIN_RAW` domain.
Install debug hooks:
@@ -531,5 +531,11 @@ Changes in the Python API
Changes in the C API
--------------------
* :c:func:`PyMem_Malloc` allocator family now uses the :ref:`pymalloc allocator
<pymalloc>` rather than system :c:func:`malloc`. Applications calling
:c:func:`PyMem_Malloc` without holding the GIL can now crash. Set the
:envvar:`PYTHONMALLOC` environment variable to ``debug`` to validate the
usage of memory allocators in your application. See :issue:`26249`.
* :c:func:`Py_Exit` (and the main interpreter) now override the exit status
with 120 if flushing buffered data failed. See :issue:`5319`.
@@ -10,6 +10,13 @@ Release date: tba
Core and Builtins
-----------------
- Issue #26249: Memory functions of the :c:func:`PyMem_Malloc` domain
(:c:data:`PYMEM_DOMAIN_MEM`) now use the :ref:`pymalloc allocator <pymalloc>`
rather than system :c:func:`malloc`. Applications calling
:c:func:`PyMem_Malloc` without holding the GIL can now crash: use
``PYTHONMALLOC=debug`` environment variable to validate the usage of memory
allocators in your application.
- Issue #26802: Optimize function calls only using unpacking like
``func(*tuple)`` (no other positional argument, no keyword): avoid copying
the tuple. Patch written by Joe Jevnik.
@@ -198,9 +198,9 @@ static PyMemAllocatorEx _PyMem_Raw = {
static PyMemAllocatorEx _PyMem = {
#ifdef Py_DEBUG
&_PyMem_Debug.mem, PYDBG_FUNCS
&_PyMem_Debug.obj, PYDBG_FUNCS
#else
NULL, PYMEM_FUNCS
NULL, PYOBJ_FUNCS
#endif
};
@@ -256,7 +256,7 @@ _PyMem_SetupAllocators(const char *opt)
PyMemAllocatorEx obj_alloc = {NULL, PYOBJ_FUNCS};
PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &mem_alloc);
PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &mem_alloc);
PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &obj_alloc);
PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &obj_alloc);
if (strcmp(opt, "pymalloc_debug") == 0)

0 comments on commit f5c4b99

Please sign in to comment.