Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ResourceWarning: Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted #70754

Closed
vstinner opened this issue Mar 15, 2016 · 11 comments

Comments

@vstinner
Copy link
Member

vstinner commented Mar 15, 2016

BPO 26567
Nosy @vstinner, @serhiy-storchaka, @1st1
Files
  • warnings_tracemalloc.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2016-03-20.11:25:42.765>
    created_at = <Date 2016-03-15.13:15:16.275>
    labels = []
    title = 'ResourceWarning: Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted'
    updated_at = <Date 2016-03-20.11:25:42.764>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2016-03-20.11:25:42.764>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2016-03-20.11:25:42.765>
    closer = 'vstinner'
    components = []
    creation = <Date 2016-03-15.13:15:16.275>
    creator = 'vstinner'
    dependencies = []
    files = ['42171']
    hgrepos = []
    issue_num = 26567
    keywords = ['patch']
    message_count = 11.0
    messages = ['261812', '261815', '261907', '262007', '262012', '262013', '262015', '262016', '262029', '262030', '262067']
    nosy_count = 4.0
    nosy_names = ['vstinner', 'python-dev', 'serhiy.storchaka', 'yselivanov']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = None
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue26567'
    versions = ['Python 3.6']

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 15, 2016

    Python emits ResourceWarning when an object using limited resource is destroyed without being explicitly closed: files, sockets, etc. The problem is that it's hard to find where the object comes from, since the warning can occur very late, in the garbage collector, etc.

    I propose to reuse tracemalloc.get_object_traceback() to show were the object was allocated, when tracemalloc traces memory allocations. In practice, I propose to add a new "source" parameter to warnings.showwarning().

    Attached patch:

    • Add a new keyword-only source parameter to warnings.showwarning()
    • Add C function PyErr_ResourceWarning() to pass source
    • showwarning() uses tracemalloc.get_object_traceback() to get the traceback were the object was allocated
    • Modify socket.socket, io.FileIO and os.scandir destructor to use PyErr_ResourceWarning()

    Backward-compatibility problem: The C function PyErr_ResourceWarning() always call warnings.showwarning() with the keyword parameter source. If an application replaces the warnings.showwarning() function, it will probably fail because it doesn't know the source parameter.

    I don't know how to handle this backward compatibility issue.

    The patch is incomplete, it's not possible yet to emit a warning in pure Python with a source parameter.

    x.py script used for examples below:
    -----------------

    import warnings
    import os
    import socket
    
    def func2():
        #f=open("/etc/issue")
        #f=os.scandir('.')
        f=socket.socket()
        f=None
    
    def func():
        func2()
    
    func()

    Output with Python 3.5:
    -----
    x.py:9: ResourceWarning: unclosed <socket.socket fd=3, ...>
    f=None
    -----

    Output with -X tracemalloc=5 command line option and patched Python 3.6:
    -----
    x.py:9: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>
    f=None
    Object allocated at (most recent call first):
    File "x.py", lineno 8
    f=socket.socket()
    File "x.py", lineno 12
    func2()
    File "x.py", lineno 14
    func()
    -----

    It's much easier to understand where the warning comes from, no? At x.py:8, line "f=socket.socket()".

    Note: the traceback doesn't contain the function name, since tracemalloc only stores filename and line number.

    See also the issue bpo-26564 "Malloc debug hooks: display memory block traceback on error".

    For Python < 3.6, I wrote "res_warn.py" script which monkey-patches io.FileIO and socket.socket to implement something similar. The script is a fragile hack. I would prefer to have the feature built-in Python.

    https://bitbucket.org/haypo/misc/src/0a40f27360424145bad0f9b62c9e9148ffdbb169/python/res_warn.py

    @vstinner vstinner changed the title Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted ResourceWarning: Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted Mar 15, 2016
    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 15, 2016

    Backward-compatibility problem: The C function PyErr_ResourceWarning() always call warnings.showwarning() with the keyword parameter source. If an application replaces the warnings.showwarning() function, it will probably fail because it doesn't know the source parameter.

    I proposed the issue bpo-26568 "Add a new warnings.showmsg() function taking a warnings.WarningMessage object" to fix this issue.

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 17, 2016

    I used this patch to identify a ResourceWarning in test_faulthandler when the test is interrupted by CTRL+c: it's really efficient!

    I will be very useful to identify all ResourceWarning that I saw in test_asyncio, like the ones seen on the AIX buildbot.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 19, 2016

    New changeset 2428d794b0e1 by Victor Stinner in branch 'default':
    On ResourceWarning, log traceback where the object was allocated
    https://hg.python.org/cpython/rev/2428d794b0e1

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 19, 2016

    It looks like io.FileIO has a strong implementation of the destructor. If the object becomes alive again because of random code called in the destructor, the object is not removed.

    socket and os.scandir have a classical unsafe destructor.

    Moreover, I'm no more sure about the chosen design. When warnings.catch_warnings() is used and an unclosed io.FileIO is destroyed, the object is kept alive because it is stored in the "source" attribute of a warnings.WarningMessage.

    I don't know if keeping WarningMessaging alive longer than the call to showwarning() (or _showarnmsg) is a common use case or not. The issue bpo-26568 wants to promote the WarningMessage class, so some users may start to keep it alive.

    An alternative is to format the object traceback and pass the traceback to WarningMessage. It requires to decide the format of the traceback (list of ..., string, something else?).

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 19, 2016

    When warnings._showwarnmsg(), the io.FileIO object is not closed yet, so all attributes are accessible, which can be useful. Hopefully, the file is closed even if it is kept alive by the warning logger. So maybe it's ok to keep the Python object alive, if the underlying resource (the file descriptor) is released. I mean, it's not a big deal.

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 19, 2016

    For socket & scandir, I opened the issue bpo-26590: "socket destructor: implement finalizer".

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 19, 2016

    New changeset b8acf98beca9 by Victor Stinner in branch 'default':
    Try to fix test_warnings on Windows
    https://hg.python.org/cpython/rev/b8acf98beca9

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 19, 2016

    New changeset 543639cdfdb9 by Victor Stinner in branch 'default':
    Try again to fix test_warnings on Windows
    https://hg.python.org/cpython/rev/543639cdfdb9

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 19, 2016

    New changeset 691fba640266 by Victor Stinner in branch 'default':
    Issue bpo-26567: enhance ResourceWarning example
    https://hg.python.org/cpython/rev/691fba640266

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 20, 2016

    Cool, test_warnings, test_logging, etc. pass again on Windows buildbots. I close the issue.

    As I wrote, I opened a second issue for socket & os.scandir.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    None yet
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant