Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Malloc debug hooks: display memory block traceback on error #70751

Closed
vstinner opened this issue Mar 15, 2016 · 18 comments
Closed

Malloc debug hooks: display memory block traceback on error #70751

vstinner opened this issue Mar 15, 2016 · 18 comments
Labels
type-feature A feature request or enhancement

Comments

@vstinner
Copy link
Member

vstinner commented Mar 15, 2016

BPO 26564
Nosy @vstinner, @vadmium, @serhiy-storchaka, @miss-islington, @iritkatriel
PRs
  • bpo-26564: fix out of date comment in traceback.c #23819
  • [3.9] bpo-26564: fix obsolete comment in traceback.c (GH-23819) #23820
  • [3.8] bpo-26564: fix obsolete comment in traceback.c (GH-23819) #23821
  • Files
  • pymem_tb.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-12-17.13:20:48.755>
    created_at = <Date 2016-03-15.00:39:55.992>
    labels = ['type-feature']
    title = 'Malloc debug hooks: display memory block traceback on error'
    updated_at = <Date 2020-12-17.13:58:11.971>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2020-12-17.13:58:11.971>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-12-17.13:20:48.755>
    closer = 'iritkatriel'
    components = []
    creation = <Date 2016-03-15.00:39:55.992>
    creator = 'vstinner'
    dependencies = []
    files = ['42167']
    hgrepos = []
    issue_num = 26564
    keywords = ['patch']
    message_count = 18.0
    messages = ['261793', '261813', '261816', '261827', '261828', '261829', '261830', '261841', '261845', '261846', '261849', '261956', '383236', '383238', '383241', '383242', '383244', '383246']
    nosy_count = 6.0
    nosy_names = ['vstinner', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'miss-islington', 'iritkatriel']
    pr_nums = ['23819', '23820', '23821']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue26564'
    versions = ['Python 3.6']

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 15, 2016

    Python implements debug hooks on Python memory allocators:
    https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks

    Problem: buffer understand and buffer overflow are only detected when a memory block is released, which may occur far from the code responsible to create the buffer.

    Attached patch dumps the traceback where a memory block was allocated on error in malloc debug hooks. The feature requires to enable tracemalloc to record tracebacks.

    test.py used in below example:
    ----

    import _testcapi
    
    def f():
        _testcapi.pymem_buffer_overflow()
    
    f()

    Example:
    ---
    haypo@selma$ ./python -X tracemalloc=5 test.py
    Debug memory block at address p=0x22e8be0: API 'm'
    16 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x22e8bf0 are not all FORBIDDENBYTE (0xfb):
    at tail+0: 0x78 *** OUCH
    ...
    The block was made by call bpo-37240 to debug malloc/realloc.
    Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb

    Memory block traceback (most recent call first):
    File "test.py", line 4
    File "test.py", line 6

    Fatal Python error: bad trailing pad byte

    Current thread 0x00007f4a93d9c700 (most recent call first):
    File "test.py", line 4 in f
    File "test.py", line 6 in <module>
    Abandon (core dumped)
    ---

    The patch adds the "Memory block traceback" section.

    @vstinner vstinner added the type-feature A feature request or enhancement label Mar 15, 2016
    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 15, 2016

    See also the issue bpo-26567: "Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted".

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 15, 2016

    I reviewed my own patch on Rietveld :-)

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 15, 2016

    New changeset 18a19e62bac5 by Victor Stinner in branch 'default':
    Enhance and rewrite traceback dump C functions
    https://hg.python.org/cpython/rev/18a19e62bac5

    New changeset fea3c6e9a38e by Victor Stinner in branch '3.5':
    _tracemalloc: store lineno as unsigned int
    https://hg.python.org/cpython/rev/fea3c6e9a38e

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 15, 2016

    New changeset cef6a32d805f by Victor Stinner in branch 'default':
    On memory error, dump the memory block traceback
    https://hg.python.org/cpython/rev/cef6a32d805f

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 15, 2016

    New changeset 8215dae7ec3c by Victor Stinner in branch 'default':
    Oops, revert unwanted change used to create an example
    https://hg.python.org/cpython/rev/8215dae7ec3c

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 15, 2016

    New changeset 769dfcb701ee by Victor Stinner in branch 'default':
    Issue bpo-26564: Fix test_capi
    https://hg.python.org/cpython/rev/769dfcb701ee

    @vadmium
    Copy link
    Member

    vadmium commented Mar 16, 2016

    FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array:

    ..\Python\traceback.c(513): error C2057: expected constant expression [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
    ..\Python\traceback.c(513): error C2466: cannot allocate an array of constant size 0 [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
    ..\Python\traceback.c(513): error C2133: 'buffer': unknown size [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 16, 2016

    New changeset 5f2284ecf9c6 by Victor Stinner in branch 'default':
    Fix compilation error of traceback.c on Windows
    https://hg.python.org/cpython/rev/5f2284ecf9c6

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 16, 2016

    Martin Panter added the comment:

    FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array:

    Thank you, sometimes I miss buildbot failures.

    I pushed changes which had 2 tests, so I knew that all buildbots were
    red, and I planned to come back the day after to check.

    For Windows, it's a pity, the expression is constant. Stupid compiler
    :-) VS2015 doesn't support C99 yet? Anyway, it's trivial to fix the
    code. I just fixed it.

    I will check again if buildbots are now happy with the current code ;-)

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 16, 2016

    Ok, buildbots are green again. I close the issue.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 18, 2016

    New changeset 7b079adb0774 by Victor Stinner in branch 'default':
    Enhance documentation on malloc debug hooks
    https://hg.python.org/cpython/rev/7b079adb0774

    @iritkatriel
    Copy link
    Member

    iritkatriel commented Dec 17, 2020

    I'm reopening this to commit a fix for a comment which became obsolete following this change: https://hg.python.org/cpython/rev/18a19e62bac5.

    @iritkatriel iritkatriel reopened this Dec 17, 2020
    @vstinner
    Copy link
    Member Author

    vstinner commented Dec 17, 2020

    New changeset 40125ab by Irit Katriel in branch 'master':
    bpo-26564: fix obsolete comment in traceback.c (GH-23819)
    40125ab

    @miss-islington
    Copy link
    Contributor

    miss-islington commented Dec 17, 2020

    New changeset cecbaa3 by Miss Islington (bot) in branch '3.8':
    bpo-26564: fix obsolete comment in traceback.c (GH-23819)
    cecbaa3

    @miss-islington
    Copy link
    Contributor

    miss-islington commented Dec 17, 2020

    New changeset 1c70d40 by Miss Islington (bot) in branch '3.9':
    bpo-26564: fix obsolete comment in traceback.c (GH-23819)
    1c70d40

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Dec 17, 2020

    I am wondering why 53/22? 5/2 is as good for 8-bytes long -- both gives 21-bytes buffer.

    I do not propose to change anything. It is just curiosity.

    @vstinner
    Copy link
    Member Author

    vstinner commented Dec 17, 2020

    I am wondering why 53/22? 5/2 is as good for 8-bytes long -- both gives 21-bytes buffer.

    53/22 is closer to math.log(256)/math.log(10) than 5/2, and 8*53 should not overflow :-)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants