-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add PYTHONMALLOC env var and add support for malloc debug hooks in release mode #70703
Comments
Attached patch:
PYTHONMALLOC is used even if -E command line option is used, I prefer to keep the code simple. PYTHONMALLOC must be checked before the first call to any Python memory allocator, so it must occur very earlier. Well, it's simply the first instruction of main()... My main use case is to be able to use debug hooks to detect:
|
The output of sys._debugmallocstats() contains different info:
The available info depends on:
|
Example with Python compiled in release mode. By default, a buffer overflow is not detected. It may crash later, in a random place... $ ./python -c 'import _testcapi; _testcapi.pymem_buffer_overflow()' Enabling debug hooks detects the buffer overflow immediatly: $ PYTHONMALLOC=pymalloc_debug ./python -c 'import _testcapi; _testcapi.pymem_buffer_overflow()'
Debug memory block at address p=0x1a7f490: API 'm'
16 bytes originally requested
The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
The 8 pad bytes at tail=0x1a7f4a0 are not all FORBIDDENBYTE (0xfb):
at tail+0: 0x78 *** OUCH
at tail+1: 0xfb
at tail+2: 0xfb
at tail+3: 0xfb
at tail+4: 0xfb
at tail+5: 0xfb
at tail+6: 0xfb
at tail+7: 0xfb
The block was made by call python/issues-test-cpython#35014 to debug malloc/realloc.
Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb
Fatal Python error: bad trailing pad byte Current thread 0x00007fca30572700 (most recent call first): |
The motivation to support PYTHONMALLOC=malloc (always use malloc) and not just PYTHONMALLOC=debug (enable debug hooks) is to allow to use external memory debugger like Valgrind. Valgrind doesn't like pymalloc (pymalloc raises false alarms), we had to a configuration option for it: ./configure --with-valgrind. Using PYTHONMALLOC=malloc allows to use Valgrind on the system Python, it's easier to use. No need to recompile Python. |
New changeset af2fc10f703b by Victor Stinner in branch '3.5': |
More complete patch (version 2):
|
When I wrote the PEP-445, I already proposed to add a new environment variable to install debug hooks on a Python compiled in release mode: https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable But the proposition was rejected because of the PEP-432, but 3 years later, this PEP is still stuck at the draft state. So I propose to move on. Anyway, PYTHONMALLOC is really a corner case since it must be read *very early* (first instruction of main()). Oh by the way, I forgot to explain the initial motivation for this change! It's the issue bpo-26249: "Change PyMem_Malloc to use pymalloc allocator". With this change, applications misuing PyMem_Malloc API (ex: alloc memory with PyMem_Malloc(), free memory with free()) will start to crash. PYTHONMALLOC=debug should help these applications to detect bugs easier and get an obvious error message. |
Patch 3:
|
Oh, I forgot to update wmain() for Windows. Patch 4 handles wmain() too. I added a few comments and reformatted some parts of the code. |
New changeset aa280432e9c7 by Victor Stinner in branch 'default': |
I reworked my patch before pushing it:
|
Note: I just checked PYTHONMALLOCSTATS=1 with -X tracemalloc. It looks like it works as expected. Tracemalloc hooks are unregistered before stats are displayed at exit, _Pymem_PymallocEnabled() returns 1 as expected. |
test_capi fails on Windows: Probably a issue with newline characters. |
New changeset 3c3df33f2655 by Victor Stinner in branch 'default': |
New changeset d8d6ec1333e6 by Victor Stinner in branch 'default': |
New changeset 7534eb7bd57e by Victor Stinner in branch 'default': |
Buildbots are happy, I close the issue. |
New changeset 7b079adb0774 by Victor Stinner in branch 'default': |
New changeset c4c14e34e528 by Victor Stinner in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: