Captured exceptions are keeping user objects alive unnecessarily in the stdlib

[mariocj89]

BPO	36820
Nosy	@Yhg1s, @warsaw, @ericvw, @mariocj89, @pablogsal, @miss-islington
PRs	bpo-36820: Break unnecessary cycle in socket.py, codeop.py and dyld.py #13135 [3.7] bpo-36820: Break unnecessary cycle in socket.py, codeop.py and dyld.py (GH-13135) #17485 [3.8] bpo-36820: Break unnecessary cycle in socket.py, codeop.py and dyld.py (GH-13135) #17486

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-12-06.14:49:38.004>
created_at = <Date 2019-05-06.19:32:15.480>
labels = ['3.8', 'type-feature', 'library']
title = 'Captured exceptions are keeping user objects alive unnecessarily in the stdlib'
updated_at = <Date 2019-12-06.15:01:35.177>
user = 'https://github.com/mariocj89'

bugs.python.org fields:

activity = <Date 2019-12-06.15:01:35.177>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2019-12-06.14:49:38.004>
closer = 'pablogsal'
components = ['Library (Lib)']
creation = <Date 2019-05-06.19:32:15.480>
creator = 'mariocj89'
dependencies = []
files = []
hgrepos = []
issue_num = 36820
keywords = ['patch']
message_count = 4.0
messages = ['341620', '357916', '357917', '357918']
nosy_count = 6.0
nosy_names = ['twouters', 'barry', 'ericvw', 'mariocj89', 'pablogsal', 'miss-islington']
pr_nums = ['13135', '17485', '17486']
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue36820'
versions = ['Python 3.8']

[mariocj89]

There are multiple places in the standard library where the code captures an exception and reraises it later (outside of the original except).

This is known to cause a cycle as saving the exception has a traceback that eventually points back to the exception, but it is especially problematic as it keeps alive the objects that the user has as well.

This can be reproduced with the following example:

import weakref

class A: pass

ref = None

def x():
    global ref
    cool_var = A()
    ref = weakref.ref(cool_var)
    assert ref()
    try:
        1/0
    except Exception as e:
        ee = e

try:
    x()
except Exception:
    pass

print(ref())
assert ref() is None

There are places in the standard library that try to get around this. See acb9fa7 as an example. This change did not include the exception path though, when the err variable is raised, the issue is still present.

This issue is to fix the instances found in socket.py, codeop.py and dyld.py. By either setting the variable to None to remove the name or if it needs to be reraised by using a finally to delete it. This is basically the same that a try except would do, which automagically adds a finally to delete the local name used in the except X as y.
There was a discussion with twouters,pablogsal,barry about potentially adding something extra to the exception handling to try to clean this up but it was suggested the best approach is for the user to handle it and maybe a linter to potentially flag it, as there might also be multiple references to the exception being captured. It was also proposed to just change the situations where this happens in the standard library to remove the name if possible.

Note that eventually those objects will just be collected by the gc, it is just an improvement.

I'm preparing a PR for those three modules.

[miss-islington]

New changeset b64334c by Miss Islington (bot) (Mario Corchero) in branch 'master':
bpo-36820: Break unnecessary cycle in socket.py, codeop.py and dyld.py (GH-13135)
b64334c

[miss-islington]

New changeset 681285d by Miss Islington (bot) in branch '3.8':
bpo-36820: Break unnecessary cycle in socket.py, codeop.py and dyld.py (GH-13135)
681285d

[miss-islington]

New changeset 5ba591f by Miss Islington (bot) in branch '3.7':
bpo-36820: Break unnecessary cycle in socket.py, codeop.py and dyld.py (GH-13135)
5ba591f