test_io leaks references on AMD64 Fedora Rawhide Refleaks 3.8

[vstinner]

BPO	37261
Nosy	@vstinner, @miss-islington
PRs	bpo-37261: Fix support.catch_unraisable_exception() #14052 [3.8] bpo-37261: Fix support.catch_unraisable_exception() (GH-14052) #14056 bpo-37261: Document sys.unraisablehook corner cases #14059 [3.8] bpo-37261: Document sys.unraisablehook corner cases (GH-14059) #14089

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-13.13:45:26.159>
created_at = <Date 2019-06-13.01:43:10.730>
labels = ['3.8']
title = 'test_io leaks references on AMD64 Fedora Rawhide Refleaks 3.8'
updated_at = <Date 2019-06-14.16:59:57.981>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-06-14.16:59:57.981>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-13.13:45:26.159>
closer = 'vstinner'
components = []
creation = <Date 2019-06-13.01:43:10.730>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 37261
keywords = ['patch']
message_count = 7.0
messages = ['345449', '345450', '345514', '345516', '345518', '345604', '345614']
nosy_count = 2.0
nosy_names = ['vstinner', 'miss-islington']
pr_nums = ['14052', '14056', '14059', '14089']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue37261'
versions = ['Python 3.8']

[vstinner]

test_io leaks references on AMD64 Fedora Rawhide Refleaks 3.8:
https://buildbot.python.org/all/#/builders/229/builds/10

test_io leaked [23208, 23204, 23208] references, sum=69620
test_io leaked [7657, 7655, 7657] memory blocks, sum=22969

The issue has been introduced by my change:

commit c15a682
Author: Victor Stinner <vstinner@redhat.com>
Date: Thu Jun 13 00:23:49 2019 +0200

bpo-37223: test_io: silence destructor errors (GH-14031)

* bpo-18748: Fix _pyio.IOBase destructor (closed case) (GH-13952)

_pyio.IOBase destructor now does nothing if getting the closed
attribute fails to better mimick _io.IOBase finalizer.

(cherry picked from commit 4f6f7c5a611905fb6b81671547f268c226bc646a)

* bpo-37223: test_io: silence destructor errors (GH-13954)

Implement also MockNonBlockWriterIO.seek() method.

(cherry picked from commit b589cef9c4dada2fb84ce0fae5040ecf16d9d5ef)

* bpo-37223, test_io: silence last 'Exception ignored in:' (GH-14029)

Use catch_unraisable_exception() to ignore 'Exception ignored in:'
error when the internal BufferedWriter of the BufferedRWPair is
destroyed. The C implementation doesn't give access to the
internal BufferedWriter, so just ignore the warning instead.

(cherry picked from commit 913fa1c8245d1cde6edb4254f4fb965cc91786ef)

It seems like the root issue is the usage of catch_unraisable_exception() in test_io.

class catch_unraisable_exception:
    def __init__(self):
        self.unraisable = None
        self._old_hook = None

    def _hook(self, unraisable):
        self.unraisable = unraisable

    def __enter__(self):
        self._old_hook = sys.unraisablehook
        sys.unraisablehook = self._hook
        return self

    def __exit__(self, *exc_info):
        # Clear the unraisable exception to explicitly break a reference cycle
        del self.unraisable
        sys.unraisablehook = self._old_hook

*Sometimes* "del self.unraisable" of __exit__() triggers a new unraisable exception which calls again the _hook() method which sets again the unraisable attribute.

catch_unraisable_exception resurects _io.BufferedWriter objects, and then "del self.unraisable" calls again their finalizer: iobase_finalize() is called again, and iobase_finalize() calls PyErr_WriteUnraisable() on close() failure.

[vstinner]

gdb traceback on "del self.unraisable":

(gdb) where
#0 _PyErr_WriteUnraisableDefaultHook (
args=(<type at remote 0x7a9fa0>, ValueError('flush of closed file',), <traceback at remote 0x7fffe9cfb5f0>, None, <_io.BufferedWriter at remote 0x7fffe9bcb890>)) at Python/errors.c:1294
#1 0x00000000005726ec in sys_unraisablehook (module=<module at remote 0x7fffeac10710>,
unraisable=(<type at remote 0x7a9fa0>, ValueError('flush of closed file',), <traceback at remote 0x7fffe9cfb5f0>, None, <_io.BufferedWriter at remote 0x7fffe9bcb890>)) at ./Python/sysmodule.c:702
#2 0x0000000000435741 in _PyMethodDef_RawFastCallKeywords (method=0x7bb2e0 <sys_methods+1216>, self=<module at remote 0x7fffeac10710>, args=0x7ffffffd1d00,
nargs=1, kwnames=0x0) at Objects/call.c:650
#3 0x00000000004359f4 in _PyCFunction_Vectorcall (func=<built-in method unraisablehook of module object at remote 0x7fffeac10710>, args=0x7ffffffd1d00,
nargsf=1, kwnames=0x0) at Objects/call.c:737
#4 0x0000000000534666 in _PyObject_Vectorcall (callable=<built-in method unraisablehook of module object at remote 0x7fffeac10710>, args=0x7ffffffd1d00,
nargsf=1, kwnames=0x0) at ./Include/cpython/abstract.h:127
#5 0x00000000005346b1 in _PyObject_FastCall (func=<built-in method unraisablehook of module object at remote 0x7fffeac10710>, args=0x7ffffffd1d00, nargs=1)
at ./Include/cpython/abstract.h:147
#6 0x0000000000536da4 in _PyErr_WriteUnraisableMsg (err_msg_str=0x0, obj=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at Python/errors.c:1379
#7 0x0000000000536ee6 in PyErr_WriteUnraisable (obj=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at Python/errors.c:1425

#8 0x00000000005eb97d in iobase_finalize (self=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at ./Modules/_io/iobase.c:299
#9 0x0000000000479fd2 in PyObject_CallFinalizer (self=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at Objects/object.c:307
#10 0x000000000047a044 in PyObject_CallFinalizerFromDealloc (self=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at Objects/object.c:325
#11 0x00000000005eb9e0 in _PyIOBase_finalize (self=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at ./Modules/_io/iobase.c:320

#12 0x00000000005f27d3 in buffered_dealloc (self=0x7fffe9bcb890) at ./Modules/_io/bufferedio.c:387
#13 0x000000000047ebff in _Py_Dealloc (op=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at Objects/object.c:2213
#14 0x00000000005f210a in _Py_DECREF (filename=0x71eef3 "./Modules/_io/bufferedio.c", lineno=2138, op=<_io.BufferedWriter at remote 0x7fffe9bcb890>)
at ./Include/object.h:478
#15 0x00000000005f716a in bufferedrwpair_dealloc (self=0x7fffe9a90d70) at ./Modules/_io/bufferedio.c:2138

#16 0x000000000047ebff in _Py_Dealloc (op=<_io.BufferedRWPair at remote 0x7fffe9a90d70>) at Objects/object.c:2213
#17 0x000000000048d1b6 in _Py_DECREF (filename=0x69fe40 "./Include/object.h", lineno=541, op=<_io.BufferedRWPair at remote 0x7fffe9a90d70>)
at ./Include/object.h:478
#18 0x000000000048d1e2 in _Py_XDECREF (op=<_io.BufferedRWPair at remote 0x7fffe9a90d70>) at ./Include/object.h:541

# del UnraisableHookArgs

#19 0x000000000048d569 in structseq_dealloc (obj=0x7fffe9ad1360) at Objects/structseq.c:85
#20 0x000000000047ebff in _Py_Dealloc (
    op=(<type at remote 0x7a9fa0>, ValueError('flush of closed file',), <unknown at remote 0x7fffe9a8ad70>, None, <_io.BufferedRWPair at remote 0x7fffe9a90d70>))
    at Objects/object.c:2213
#21 0x0000000000463be6 in _Py_DECREF (filename=0x696dc9 "Objects/dictobject.c", lineno=1585, 
    op=(<type at remote 0x7a9fa0>, ValueError('flush of closed file',), <unknown at remote 0x7fffe9a8ad70>, None, <_io.BufferedRWPair at remote 0x7fffe9a90d70>))
    at ./Include/object.h:478
#22 0x0000000000468428 in delitem_common (mp=0x7fffe9b17950, hash=-822296730167156087, ix=0, 
    old_value=(<type at remote 0x7a9fa0>, ValueError('flush of closed file',), <unknown at remote 0x7fffe9a8ad70>, None, <_io.BufferedRWPair at remote 0x7fffe9a90d70>)) at Objects/dictobject.c:1585
#23 0x0000000000468663 in _PyDict_DelItem_KnownHash (op={'_old_hook': <built-in method unraisablehook of module object at remote 0x7fffeac10710>}, 
    key='unraisable', hash=-822296730167156087) at Objects/dictobject.c:1637
#24 0x00000000004684e5 in PyDict_DelItem (op={'_old_hook': <built-in method unraisablehook of module object at remote 0x7fffeac10710>}, key='unraisable')
    at Objects/dictobject.c:1603

# del self.unraisable

#25 0x000000000046ef21 in _PyObjectDict_SetItem (tp=0xa58830, dictptr=0x7fffe9a8ad30, key='unraisable', value=0x0) at Objects/dictobject.c:4614

[vstinner]

New changeset 6d22cc8 by Victor Stinner in branch 'master':
bpo-37261: Fix support.catch_unraisable_exception() (GH-14052)
6d22cc8

[miss-islington]

New changeset b4f5b21 by Miss Islington (bot) in branch '3.8':
bpo-37261: Fix support.catch_unraisable_exception() (GH-14052)
b4f5b21

[vstinner]

I tested manually "./python -m test test_io -m CBufferedRWPairTest -R 3:3": it failed before the fix, it now pass on 3.8 and master with the fix. I close the issue.

Enjoy resurrected objects...

[vstinner]

New changeset 212646c by Victor Stinner in branch 'master':
bpo-37261: Document sys.unraisablehook corner cases (GH-14059)
212646c

[miss-islington]

New changeset 3b976d1 by Miss Islington (bot) in branch '3.8':
bpo-37261: Document sys.unraisablehook corner cases (GH-14059)
3b976d1