ResourceWarning: Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted #70754
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
assignee = None closed_at = <Date 2016-03-20.11:25:42.765> created_at = <Date 2016-03-15.13:15:16.275> labels =  title = 'ResourceWarning: Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted' updated_at = <Date 2016-03-20.11:25:42.764> user = 'https://github.com/vstinner'
activity = <Date 2016-03-20.11:25:42.764> actor = 'vstinner' assignee = 'none' closed = True closed_date = <Date 2016-03-20.11:25:42.765> closer = 'vstinner' components =  creation = <Date 2016-03-15.13:15:16.275> creator = 'vstinner' dependencies =  files = ['42171'] hgrepos =  issue_num = 26567 keywords = ['patch'] message_count = 11.0 messages = ['261812', '261815', '261907', '262007', '262012', '262013', '262015', '262016', '262029', '262030', '262067'] nosy_count = 4.0 nosy_names = ['vstinner', 'python-dev', 'serhiy.storchaka', 'yselivanov'] pr_nums =  priority = 'normal' resolution = 'fixed' stage = None status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue26567' versions = ['Python 3.6']
The text was updated successfully, but these errors were encountered:
Python emits ResourceWarning when an object using limited resource is destroyed without being explicitly closed: files, sockets, etc. The problem is that it's hard to find where the object comes from, since the warning can occur very late, in the garbage collector, etc.
I propose to reuse tracemalloc.get_object_traceback() to show were the object was allocated, when tracemalloc traces memory allocations. In practice, I propose to add a new "source" parameter to warnings.showwarning().
Backward-compatibility problem: The C function PyErr_ResourceWarning() always call warnings.showwarning() with the keyword parameter source. If an application replaces the warnings.showwarning() function, it will probably fail because it doesn't know the source parameter.
I don't know how to handle this backward compatibility issue.
The patch is incomplete, it's not possible yet to emit a warning in pure Python with a source parameter.
x.py script used for examples below:
import warnings import os import socket def func2(): #f=open("/etc/issue") #f=os.scandir('.') f=socket.socket() f=None def func(): func2() func()
Output with Python 3.5:
Output with -X tracemalloc=5 command line option and patched Python 3.6:
It's much easier to understand where the warning comes from, no? At x.py:8, line "f=socket.socket()".
Note: the traceback doesn't contain the function name, since tracemalloc only stores filename and line number.
See also the issue bpo-26564 "Malloc debug hooks: display memory block traceback on error".
For Python < 3.6, I wrote "res_warn.py" script which monkey-patches io.FileIO and socket.socket to implement something similar. The script is a fragile hack. I would prefer to have the feature built-in Python.
I proposed the issue bpo-26568 "Add a new warnings.showmsg() function taking a warnings.WarningMessage object" to fix this issue.
It looks like io.FileIO has a strong implementation of the destructor. If the object becomes alive again because of random code called in the destructor, the object is not removed.
socket and os.scandir have a classical unsafe destructor.
Moreover, I'm no more sure about the chosen design. When warnings.catch_warnings() is used and an unclosed io.FileIO is destroyed, the object is kept alive because it is stored in the "source" attribute of a warnings.WarningMessage.
I don't know if keeping WarningMessaging alive longer than the call to showwarning() (or _showarnmsg) is a common use case or not. The issue bpo-26568 wants to promote the WarningMessage class, so some users may start to keep it alive.
An alternative is to format the object traceback and pass the traceback to WarningMessage. It requires to decide the format of the traceback (list of ..., string, something else?).
When warnings._showwarnmsg(), the io.FileIO object is not closed yet, so all attributes are accessible, which can be useful. Hopefully, the file is closed even if it is kept alive by the warning logger. So maybe it's ok to keep the Python object alive, if the underlying resource (the file descriptor) is released. I mean, it's not a big deal.