Raised exception in Enum keeping user objects alive unnecessarily

[efroemling]

BPO	42248
Nosy	@mdickinson, @pitrou, @ethanfurman, @efroemling
PRs	bpo-42248: [Enum] ensure exceptions raised in _missing_ are released #25350 [3.8] bpo-42248: [Enum] ensure exceptions raised in _missing_ are released. #25369 [3.9] bpo-42248: [Enum] ensure exceptions raised in _missing_ are released. #25370
Files	enum_ref_loop_example.py: script to demonstrate the issue

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ethanfurman'
closed_at = <Date 2021-11-13.19:10:16.809>
created_at = <Date 2020-11-02.20:46:49.158>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'Raised exception in Enum keeping user objects alive unnecessarily'
updated_at = <Date 2021-11-13.19:10:16.809>
user = 'https://github.com/efroemling'

bugs.python.org fields:

activity = <Date 2021-11-13.19:10:16.809>
actor = 'ethan.furman'
assignee = 'ethan.furman'
closed = True
closed_date = <Date 2021-11-13.19:10:16.809>
closer = 'ethan.furman'
components = ['Library (Lib)']
creation = <Date 2020-11-02.20:46:49.158>
creator = 'efroemling'
dependencies = []
files = ['49564']
hgrepos = []
issue_num = 42248
keywords = ['patch']
message_count = 12.0
messages = ['380249', '382523', '390093', '390095', '390859', '390889', '390913', '391285', '403860', '403861', '403862', '403875']
nosy_count = 5.0
nosy_names = ['mark.dickinson', 'pitrou', 'ethan.furman', 'efroemling', 'gerald.dalley2']
pr_nums = ['25350', '25369', '25370']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42248'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[efroemling]

I've run into an issue where exceptions thrown by Enum constructors are keeping my objects alive. The underlying issue seems to be the same as https://bugs.python.org/issue36820

The same method used to fix the issue above seems to work here: simply adding a try/finally clause around the error handling at the end of enum.Enum.__new__() which sets ve_exc and exc to None does the trick.

I've attached a short script which demonstrates the issue. I realize that the cyclic garbage collector will eventually handle this case, but its a bummer to lose determinism in the destruction of my objects.

I'd be happy to create a PR for this or whatever I can do to help; just let me know if I should (I'm new here).

[geralddalley2]

I and a few others have run into issues with the Enum constructors producing spurious reference cycles. This can cause memory explosions if large objects like numpy arrays are held in any of the relevant stack frames. Based on https://bugs.python.org/issue36820, it looks like the maintainers of CPython are open to fixing similar issues, and PRs look like the way to make progress.

[mdickinson]

I'm running into this, too, on Python 3.9.2; in my case it's causing segmentation faults in a wxPython-based application.

Those segfaults aren't the fault of the reference cycle, of course; they're a result of wxPython being finicky about object cleanup order, but the existence of the enum reference cycle does make it harder to maneuver wxPython into a place where it doesn't crash.

[mdickinson]

Here's a cut-down example, at the prompt:

Python 3.9.2 (default, Mar 31 2021, 05:47:22) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import signal
>>> class App: pass
... 
>>> def create_app():
...     app = App()
...     signal.signal(signal.SIGINT, signal.SIG_DFL)
... 
>>> create_app()

At this point, since the App() instance was local to create_app, and wasn't returned, I'd expect there to be no App objects alive in the system. But it turns out there's still an App object being kept alive:

>>> import gc
>>> [obj for obj in gc.get_objects() if type(obj) is App]
[<__main__.App object at 0x10acb3d90>]

The cause is a call to _int_to_enum in signal.py which attempts to coerce the default signal handler to an element of Handlers. That coercion fails, leaving an exception

  ValueError('<built-in function default_int_handler> is not a valid Handlers')

The traceback on that exception leads to the frame containing the call to Enum.__new__, which in turn contains a reference ve_exc back to the exception.

[In the real code, App was a wx.App object, and the App.__init__ method played the role of create_app.]

[ethanfurman]

New changeset 8c14f5a by Ethan Furman in branch 'master':
bpo-42248: [Enum] ensure exceptions raised in _missing_ are released (GH-25350)
8c14f5a

[ethanfurman]

New changeset f396a1a by Ethan Furman in branch '3.8':
[3.8] bpo-42248: [Enum] ensure exceptions raised in _missing_ are released (GH-25350). (GH-25369)
f396a1a

[ethanfurman]

New changeset 6379924 by Ethan Furman in branch '3.9':
[3.9] bpo-42248: [Enum] ensure exceptions raised in _missing_ are released (GH-25350). (GH-25370)
6379924

[mdickinson]

Thank you for the quick fix!

[pitrou]

It appears this is *not* fixed in 3.10.0:

Python 3.10.0 (default, Oct 13 2021, 08:45:17) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import signal, gc
>>> class App: pass
... 
>>> def create_app():
...   app = App()
...   signal.signal(signal.SIGINT, signal.SIG_DFL)
... 
>>> [obj for obj in gc.get_objects() if type(obj) is App]
[]
>>> create_app()
>>> [obj for obj in gc.get_objects() if type(obj) is App]
[<__main__.App object at 0x7f2d3f3f1c30>]

[pitrou]

Indeed, changeset 8c14f5a is part of git main, but is not on the 3.10 branch AFAICT.

[pitrou]

It seems the problem is that bpo-44559 reset the enum module to a previous state without the bugfix.

[ethanfurman]

Going back through the various bug fixes that got cut from 3.10 to re-add to 3.10.1.