Add PYTHONMALLOC env var

Issue #26516:

* Add PYTHONMALLOC environment variable to set the Python memory
  allocators and/or install debug hooks.
* PyMem_SetupDebugHooks() can now also be used on Python compiled in release
  mode.
* The PYTHONMALLOCSTATS environment variable can now also be used on Python
  compiled in release mode. It now has no effect if set to an empty string.
* In debug mode, debug hooks are now also installed on Python memory allocators
  when Python is configured without pymalloc.

Author: vstinner

Doc/c-api/memory.rst

@@ -85,9 +85,12 @@ for the I/O buffer escapes completely the Python memory manager.
 
 .. seealso::
 
+   The :envvar:`PYTHONMALLOC` environment variable can be used to configure
+   the memory allocators used by Python.
+
    The :envvar:`PYTHONMALLOCSTATS` environment variable can be used to print
-   memory allocation statistics every time a new object arena is created, and
-   on shutdown.
+   statistics of the :ref:`pymalloc memory allocator <pymalloc>` every time a
+   new pymalloc object arena is created, and on shutdown.
 
 
 Raw Memory Interface
@@ -343,25 +346,36 @@ Customize Memory Allocators
    - detect write before the start of the buffer (buffer underflow)
    - detect write after the end of the buffer (buffer overflow)
 
-   The function does nothing if Python is not compiled is debug mode.
+   These hooks are installed by default if Python is compiled in debug
+   mode. The :envvar:`PYTHONMALLOC` environment variable can be used to install
+   debug hooks on a Python compiled in release mode.
+
+   .. versionchanged:: 3.6
+      This function now also works on Python compiled in release mode.
+
 
+.. _pymalloc:
 
-Customize PyObject Arena Allocator
-==================================
+The pymalloc allocator
+======================
 
-Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This
-allocator is optimized for small objects with a short lifetime. It uses memory
-mappings called "arenas" with a fixed size of 256 KB. It falls back to
-:c:func:`PyMem_RawMalloc` and :c:func:`PyMem_RawRealloc` for allocations larger
-than 512 bytes.  *pymalloc* is the default allocator used by
-:c:func:`PyObject_Malloc`.
+Python has a *pymalloc* allocator optimized for small objects (smaller or equal
+to 512 bytes) with a short lifetime. It uses memory mappings called "arenas"
+with a fixed size of 256 KB. It falls back to :c:func:`PyMem_RawMalloc` and
+:c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes.
 
-The default arena allocator uses the following functions:
+*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_OBJ` domain
+(:c:func:`PyObject_Malloc` & cie).
+
+The arena allocator uses the following functions:
 
 * :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows,
 * :c:func:`mmap` and :c:func:`munmap` if available,
 * :c:func:`malloc` and :c:func:`free` otherwise.
 
+Customize pymalloc Arena Allocator
+----------------------------------
+
 .. versionadded:: 3.4
 
 .. c:type:: PyObjectArenaAllocator

Doc/using/cmdline.rst

@@ -621,6 +621,51 @@ conflict.
    .. versionadded:: 3.4
 
 
+.. envvar:: PYTHONMALLOC
+
+   Set the Python memory allocators and/or install debug hooks.
+
+   Set the family of memory allocators used by Python:
+
+   * ``malloc``: use the :c:func:`malloc` function of the C library
+     for all Python memory allocators (:c:func:`PyMem_RawMalloc`,
+     :c:func:`PyMem_Malloc`, :c:func:`PyObject_Malloc` & cie).
+   * ``pymalloc``: :c:func:`PyObject_Malloc`, :c:func:`PyObject_Calloc` and
+     :c:func:`PyObject_Realloc` use the :ref:`pymalloc allocator <pymalloc>`.
+     Other Python memory allocators (:c:func:`PyMem_RawMalloc`,
+     :c:func:`PyMem_Malloc` & cie) use :c:func:`malloc`.
+
+   Install debug hooks:
+
+   * ``debug``: install debug hooks on top of the default memory allocator
+   * ``malloc_debug``: same than ``malloc`` but also install debug hooks
+   * ``pymalloc_debug``: same than ``malloc`` but also install debug hooks
+
+   See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python
+   memory allocators.
+
+   .. note::
+      ``pymalloc`` and ``pymalloc_debug`` are not available if Python is
+      configured without ``pymalloc`` support.
+
+   .. versionadded:: 3.6
+
+
+.. envvar:: PYTHONMALLOCSTATS
+
+   If set to a non-empty string, Python will print statistics of the
+   :ref:`pymalloc memory allocator <pymalloc>` every time a new pymalloc object
+   arena is created, and on shutdown.
+
+   This variable is ignored if the :envvar:`PYTHONMALLOC` environment variable
+   is used to force the :c:func:`malloc` allocator of the C library, or if
+   Python is configured without ``pymalloc`` support.
+
+   .. versionchanged:: 3.6
+      This variable can now also be used on Python compiled in release mode.
+      It now has no effect if set to an empty string.
+
+
 Debug-mode variables
 ~~~~~~~~~~~~~~~~~~~~
 
@@ -636,9 +681,3 @@ if Python was configured with the ``--with-pydebug`` build option.
 
    If set, Python will dump objects and reference counts still alive after
    shutting down the interpreter.
-
-
-.. envvar:: PYTHONMALLOCSTATS
-
-   If set, Python will print memory allocation statistics every time a new
-   object arena is created, and on shutdown.

Doc/whatsnew/3.6.rst

@@ -80,6 +80,9 @@ Summary -- Release highlights
        PEP written by Carl Meyer
 
 
+New Features
+============
+
 .. _whatsnew-fstrings:
 
 PEP 498: Formatted string literals
@@ -98,6 +101,34 @@ evaluated at run time, and then formatted using the :func:`format` protocol.
 See :pep:`498` and the main documentation at :ref:`f-strings`.
 
 
+PYTHONMALLOC environment variable
+---------------------------------
+
+The new :envvar:`PYTHONMALLOC` environment variable allows to set the Python
+memory allocators and/or install debug hooks.
+
+It is now possible to install debug hooks on Python memory allocators on Python
+compiled in release mode using ``PYTHONMALLOC=debug``. Effects of debug hooks:
+
+* Newly allocated memory is filled with the byte ``0xCB``
+* Freed memory is filled with the byte ``0xDB``
+* Detect violations of Python memory allocator API. For example,
+  :c:func:`PyObject_Free` called on a memory block allocated by
+  :c:func:`PyMem_Malloc`.
+* Detect write before the start of the buffer (buffer underflow)
+* Detect write after the end of the buffer (buffer overflow)
+
+See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python
+memory allocators.
+
+It is now also possible to force the usage of the :c:func:`malloc` allocator of
+the C library for all Python memory allocations using ``PYTHONMALLOC=malloc``.
+It helps to use external memory debuggers like Valgrind on a Python compiled in
+release mode.
+
+(Contributed by Victor Stinner in :issue:`26516`.)
+
+
 Other Language Changes
 ======================
 

Include/pymem.h

@@ -16,8 +16,17 @@ PyAPI_FUNC(void *) PyMem_RawMalloc(size_t size);
 PyAPI_FUNC(void *) PyMem_RawCalloc(size_t nelem, size_t elsize);
 PyAPI_FUNC(void *) PyMem_RawRealloc(void *ptr, size_t new_size);
 PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
+
+/* Configure the Python memory allocators. Pass NULL to use default
+   allocators. */
+PyAPI_FUNC(int) _PyMem_SetupAllocators(const char *opt);
+
+#ifdef WITH_PYMALLOC
+PyAPI_FUNC(int) _PyMem_PymallocEnabled(void);
 #endif
 
+#endif   /* !Py_LIMITED_API */
+
 
 /* BEWARE:
 

Lib/test/test_capi.py

@@ -6,6 +6,7 @@
 import random
 import subprocess
 import sys
+import sysconfig
 import textwrap
 import time
 import unittest
@@ -521,6 +522,7 @@ def test_parse_tuple_and_keywords(self):
         self.assertRaises(ValueError, _testcapi.parse_tuple_and_keywords,
                           (), {}, b'', [42])
 
+
 @unittest.skipUnless(threading, 'Threading required for this test.')
 class TestThreadState(unittest.TestCase):
 
@@ -545,6 +547,7 @@ def callback():
         t.start()
         t.join()
 
+
 class Test_testcapi(unittest.TestCase):
     def test__testcapi(self):
         for name in dir(_testcapi):
@@ -553,5 +556,61 @@ def test__testcapi(self):
                     test = getattr(_testcapi, name)
                     test()
 
+
+class MallocTests(unittest.TestCase):
+    ENV = 'debug'
+
+    def check(self, code):
+        with support.SuppressCrashReport():
+            out = assert_python_failure('-c', code, PYTHONMALLOC=self.ENV)
+        stderr = out.err
+        return stderr.decode('ascii', 'replace')
+
+    def test_buffer_overflow(self):
+        out = self.check('import _testcapi; _testcapi.pymem_buffer_overflow()')
+        regex = (r"Debug memory block at address p=0x[0-9a-f]+: API 'm'\n"
+                 r"    16 bytes originally requested\n"
+                 r"    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.\n"
+                 r"    The 8 pad bytes at tail=0x[0-9a-f]+ are not all FORBIDDENBYTE \(0x[0-9a-f]{2}\):\n"
+                 r"        at tail\+0: 0x78 \*\*\* OUCH\n"
+                 r"        at tail\+1: 0xfb\n"
+                 r"        at tail\+2: 0xfb\n"
+                 r"        at tail\+3: 0xfb\n"
+                 r"        at tail\+4: 0xfb\n"
+                 r"        at tail\+5: 0xfb\n"
+                 r"        at tail\+6: 0xfb\n"
+                 r"        at tail\+7: 0xfb\n"
+                 r"    The block was made by call #[0-9]+ to debug malloc/realloc.\n"
+                 r"    Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb\n"
+                 r"Fatal Python error: bad trailing pad byte")
+        self.assertRegex(out, regex)
+
+    def test_api_misuse(self):
+        out = self.check('import _testcapi; _testcapi.pymem_api_misuse()')
+        regex = (r"Debug memory block at address p=0x[0-9a-f]+: API 'm'\n"
+                 r"    16 bytes originally requested\n"
+                 r"    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.\n"
+                 r"    The 8 pad bytes at tail=0x[0-9a-f]+ are FORBIDDENBYTE, as expected.\n"
+                 r"    The block was made by call #[0-9]+ to debug malloc/realloc.\n"
+                 r"    Data at p: .*\n"
+                 r"Fatal Python error: bad ID: Allocated using API 'm', verified using API 'r'\n")
+        self.assertRegex(out, regex)
+
+
+class MallocDebugTests(MallocTests):
+    ENV = 'malloc_debug'
+
+
+@unittest.skipUnless(sysconfig.get_config_var('WITH_PYMALLOC') == 1,
+                     'need pymalloc')
+class PymallocDebugTests(MallocTests):
+    ENV = 'pymalloc_debug'
+
+
+@unittest.skipUnless(Py_DEBUG, 'need Py_DEBUG')
+class DefaultMallocDebugTests(MallocTests):
+    ENV = ''
+
+
 if __name__ == "__main__":
     unittest.main()

Misc/NEWS

@@ -10,6 +10,19 @@ Release date: tba
 Core and Builtins
 -----------------
 
+- Issue #26516: Add :envvar`PYTHONMALLOC` environment variable to set the
+  Python memory allocators and/or install debug hooks.
+
+- Issue #26516: The :c:func`PyMem_SetupDebugHooks` function can now also be
+  used on Python compiled in release mode.
+
+- Issue #26516: The :envvar:`PYTHONMALLOCSTATS` environment variable can now
+  also be used on Python compiled in release mode. It now has no effect if
+  set to an empty string.
+
+- Issue #26516: In debug mode, debug hooks are now also installed on Python
+  memory allocators when Python is configured without pymalloc.
+
 - Issue #26464: Fix str.translate() when string is ASCII and first replacements
   removes character, but next replacement uses a non-ASCII character or a
   string longer than 1 character. Regression introduced in Python 3.5.0.

Misc/README.valgrind

@@ -2,6 +2,9 @@ This document describes some caveats about the use of Valgrind with
 Python.  Valgrind is used periodically by Python developers to try
 to ensure there are no memory leaks or invalid memory reads/writes.
 
+UPDATE: Python 3.6 now supports PYTHONMALLOC=malloc environment variable which
+can be used to force the usage of the malloc() allocator of the C library.
+
 If you don't want to read about the details of using Valgrind, there
 are still two things you must do to suppress the warnings.  First,
 you must use a suppressions file.  One is supplied in

Modules/_testcapimodule.c

@@ -3616,6 +3616,33 @@ get_recursion_depth(PyObject *self, PyObject *args)
     return PyLong_FromLong(tstate->recursion_depth - 1);
 }
 
+static PyObject*
+pymem_buffer_overflow(PyObject *self, PyObject *args)
+{
+    char *buffer;
+
+    /* Deliberate buffer overflow to check that PyMem_Free() detects
+       the overflow when debug hooks are installed. */
+    buffer = PyMem_Malloc(16);
+    buffer[16] = 'x';
+    PyMem_Free(buffer);
+
+    Py_RETURN_NONE;
+}
+
+static PyObject*
+pymem_api_misuse(PyObject *self, PyObject *args)
+{
+    char *buffer;
+
+    /* Deliberate misusage of Python allocators:
+       allococate with PyMem but release with PyMem_Raw. */
+    buffer = PyMem_Malloc(16);
+    PyMem_RawFree(buffer);
+
+    Py_RETURN_NONE;
+}
+
 
 static PyMethodDef TestMethods[] = {
     {"raise_exception",         raise_exception,                 METH_VARARGS},
@@ -3798,6 +3825,8 @@ static PyMethodDef TestMethods[] = {
     {"PyTime_AsMilliseconds", test_PyTime_AsMilliseconds, METH_VARARGS},
     {"PyTime_AsMicroseconds", test_PyTime_AsMicroseconds, METH_VARARGS},
     {"get_recursion_depth", get_recursion_depth, METH_NOARGS},
+    {"pymem_buffer_overflow", pymem_buffer_overflow, METH_NOARGS},
+    {"pymem_api_misuse", pymem_api_misuse, METH_NOARGS},
     {NULL, NULL} /* sentinel */
 };
 

Modules/main.c

@@ -93,14 +93,15 @@ static const char usage_5[] =
 "               The default module search path uses %s.\n"
 "PYTHONCASEOK : ignore case in 'import' statements (Windows).\n"
 "PYTHONIOENCODING: Encoding[:errors] used for stdin/stdout/stderr.\n"
-"PYTHONFAULTHANDLER: dump the Python traceback on fatal errors.\n\
-";
-static const char usage_6[] = "\
-PYTHONHASHSEED: if this variable is set to 'random', a random value is used\n\
-   to seed the hashes of str, bytes and datetime objects.  It can also be\n\
-   set to an integer in the range [0,4294967295] to get hash values with a\n\
-   predictable seed.\n\
-";
+"PYTHONFAULTHANDLER: dump the Python traceback on fatal errors.\n";
+static const char usage_6[] =
+"PYTHONHASHSEED: if this variable is set to 'random', a random value is used\n"
+"   to seed the hashes of str, bytes and datetime objects.  It can also be\n"
+"   set to an integer in the range [0,4294967295] to get hash values with a\n"
+"   predictable seed.\n"
+"PYTHONMALLOC: set the Python memory allocators and/or install debug hooks\n"
+"   on Python memory allocators. Use PYTHONMALLOC=debug to install debug\n"
+"   hooks.\n";
 
 static int
 usage(int exitcode, const wchar_t* program)
@@ -341,6 +342,7 @@ Py_Main(int argc, wchar_t **argv)
     int help = 0;
     int version = 0;
     int saw_unbuffered_flag = 0;
+    char *opt;
     PyCompilerFlags cf;
     PyObject *warning_option = NULL;
     PyObject *warning_options = NULL;
@@ -365,6 +367,13 @@ Py_Main(int argc, wchar_t **argv)
         }
     }
 
+    opt = Py_GETENV("PYTHONMALLOC");
+    if (_PyMem_SetupAllocators(opt) < 0) {
+        fprintf(stderr,
+                "Error in PYTHONMALLOC: unknown allocator \"%s\"!\n", opt);
+        exit(1);
+    }
+
     Py_HashRandomizationFlag = 1;
     _PyRandom_Init();