Malloc debug hooks: display memory block traceback on error

[vstinner]

BPO	26564
Nosy	@vstinner, @vadmium, @serhiy-storchaka, @miss-islington, @iritkatriel
PRs	bpo-26564: fix out of date comment in traceback.c #23819 [3.9] bpo-26564: fix obsolete comment in traceback.c (GH-23819) #23820 [3.8] bpo-26564: fix obsolete comment in traceback.c (GH-23819) #23821
Files	pymem_tb.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-12-17.13:20:48.755>
created_at = <Date 2016-03-15.00:39:55.992>
labels = ['type-feature']
title = 'Malloc debug hooks: display memory block traceback on error'
updated_at = <Date 2020-12-17.13:58:11.971>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-12-17.13:58:11.971>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-12-17.13:20:48.755>
closer = 'iritkatriel'
components = []
creation = <Date 2016-03-15.00:39:55.992>
creator = 'vstinner'
dependencies = []
files = ['42167']
hgrepos = []
issue_num = 26564
keywords = ['patch']
message_count = 18.0
messages = ['261793', '261813', '261816', '261827', '261828', '261829', '261830', '261841', '261845', '261846', '261849', '261956', '383236', '383238', '383241', '383242', '383244', '383246']
nosy_count = 6.0
nosy_names = ['vstinner', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'miss-islington', 'iritkatriel']
pr_nums = ['23819', '23820', '23821']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26564'
versions = ['Python 3.6']

[vstinner]

Python implements debug hooks on Python memory allocators:
https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks

Problem: buffer understand and buffer overflow are only detected when a memory block is released, which may occur far from the code responsible to create the buffer.

Attached patch dumps the traceback where a memory block was allocated on error in malloc debug hooks. The feature requires to enable tracemalloc to record tracebacks.

test.py used in below example:
----

import _testcapi

def f():
    _testcapi.pymem_buffer_overflow()

f()

---

Example:
---
haypo@selma$ ./python -X tracemalloc=5 test.py
Debug memory block at address p=0x22e8be0: API 'm'
16 bytes originally requested
The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
The 8 pad bytes at tail=0x22e8bf0 are not all FORBIDDENBYTE (0xfb):
at tail+0: 0x78 *** OUCH
...
The block was made by call bpo-37240 to debug malloc/realloc.
Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb

Memory block traceback (most recent call first):
File "test.py", line 4
File "test.py", line 6

Fatal Python error: bad trailing pad byte

Current thread 0x00007f4a93d9c700 (most recent call first):
File "test.py", line 4 in f
File "test.py", line 6 in <module>
Abandon (core dumped)
---

The patch adds the "Memory block traceback" section.

[vstinner]

See also the issue bpo-26567: "Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted".

[vstinner]

I reviewed my own patch on Rietveld :-)

[python-dev]

New changeset 18a19e62bac5 by Victor Stinner in branch 'default':
Enhance and rewrite traceback dump C functions
https://hg.python.org/cpython/rev/18a19e62bac5

New changeset fea3c6e9a38e by Victor Stinner in branch '3.5':
_tracemalloc: store lineno as unsigned int
https://hg.python.org/cpython/rev/fea3c6e9a38e

[python-dev]

New changeset cef6a32d805f by Victor Stinner in branch 'default':
On memory error, dump the memory block traceback
https://hg.python.org/cpython/rev/cef6a32d805f

[python-dev]

New changeset 8215dae7ec3c by Victor Stinner in branch 'default':
Oops, revert unwanted change used to create an example
https://hg.python.org/cpython/rev/8215dae7ec3c

[python-dev]

New changeset 769dfcb701ee by Victor Stinner in branch 'default':
Issue bpo-26564: Fix test_capi
https://hg.python.org/cpython/rev/769dfcb701ee

[vadmium]

FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array:

..\Python\traceback.c(513): error C2057: expected constant expression [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
..\Python\traceback.c(513): error C2466: cannot allocate an array of constant size 0 [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
..\Python\traceback.c(513): error C2133: 'buffer': unknown size [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]

[python-dev]

New changeset 5f2284ecf9c6 by Victor Stinner in branch 'default':
Fix compilation error of traceback.c on Windows
https://hg.python.org/cpython/rev/5f2284ecf9c6

[vstinner]

Martin Panter added the comment:

FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array:

Thank you, sometimes I miss buildbot failures.

I pushed changes which had 2 tests, so I knew that all buildbots were
red, and I planned to come back the day after to check.

For Windows, it's a pity, the expression is constant. Stupid compiler
:-) VS2015 doesn't support C99 yet? Anyway, it's trivial to fix the
code. I just fixed it.

I will check again if buildbots are now happy with the current code ;-)

[vstinner]

Ok, buildbots are green again. I close the issue.

[python-dev]

New changeset 7b079adb0774 by Victor Stinner in branch 'default':
Enhance documentation on malloc debug hooks
https://hg.python.org/cpython/rev/7b079adb0774

[iritkatriel]

I'm reopening this to commit a fix for a comment which became obsolete following this change: https://hg.python.org/cpython/rev/18a19e62bac5.

[vstinner]

New changeset 40125ab by Irit Katriel in branch 'master':
bpo-26564: fix obsolete comment in traceback.c (GH-23819)
40125ab

[miss-islington]

New changeset cecbaa3 by Miss Islington (bot) in branch '3.8':
bpo-26564: fix obsolete comment in traceback.c (GH-23819)
cecbaa3

[miss-islington]

New changeset 1c70d40 by Miss Islington (bot) in branch '3.9':
bpo-26564: fix obsolete comment in traceback.c (GH-23819)
1c70d40

[serhiy-storchaka]

I am wondering why 53/22? 5/2 is as good for 8-bytes long -- both gives 21-bytes buffer.

I do not propose to change anything. It is just curiosity.

[vstinner]

I am wondering why 53/22? 5/2 is as good for 8-bytes long -- both gives 21-bytes buffer.

53/22 is closer to math.log(256)/math.log(10) than 5/2, and 8*53 should not overflow :-)