asyncio.Runner & Eventloop will try to create thread at interpreter shutdown to shutdown default executor

[fried]

Bug report

Bug description:

I'm not sure if this is a bug or not, but I ended up having to skip cleanup when sys.is_finalizing() to prevent the exception at shutdown. So maybe we skip the whole thread creation on sys.is_finalizing(), because if I shutdown the default executor before calling Runner.close() and set the default executor to None then close can take place after finalizing has begun.

import asyncio
import asyncio.constants
import time
import sys

class Magic:
	def __init__(self):
		self.runner = asyncio.Runner()

	def __del__(self):
		assert sys.is_finalizing()
		self.runner.close()

def foo():
	print("thread started")
	time.sleep(1)

x = Magic()
x.runner.get_loop().run_in_executor(None, foo)
print("going out of scope")

Results in the following output (on main branch [b706ff0])

thread started
going out of scope
Exception ignored while calling deallocator <function Magic.__del__ at 0x10480a980>:
Traceback (most recent call last):
  File "/Users/fried/projects/cpython/repro.py", line 12, in __del__
  File "/Users/fried/projects/cpython/Lib/asyncio/runners.py", line 73, in close
  File "/Users/fried/projects/cpython/Lib/asyncio/base_events.py", line 719, in run_until_complete
  File "/Users/fried/projects/cpython/Lib/asyncio/base_events.py", line 607, in shutdown_default_executor
  File "/Users/fried/projects/cpython/Lib/threading.py", line 997, in start
PythonFinalizationError: can't create new thread at interpreter shutdown

I have also seen this exception in the wild but I can't repro it.

/usr/local/lib/python3.12/asyncio/events.py:88: RuntimeWarning: The executor did not finishing joining its threads within 300 seconds.
  self._context.run(self._callback, *self._args)
Exception ignored in thread started by: <bound method Thread._bootstrap of <Thread(Thread-36 (_do_shutdown), stopped 139643360118336)>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/usr/local/lib/python3.12/threading.py", line 1079, in _bootstrap_inner
    self._delete()
  File "/usr/local/lib/python3.12/threading.py", line 1111, in _delete
    del _active[get_ident()]
        ~~~~~~~^^^^^^^^^^^^^
KeyError: 139643360118336

CPython versions tested on:

CPython main branch, 3.12

Operating systems tested on:

Linux

[kumaraditya303]

Calling asyncio APIs late in python finalization doesn't work, it relies on a lot of things which are not allowed at interpreter shutdown, it is better to rely on explicit resource management than finalizers, I don't think this is an asyncio bug.

[fried]

As far as I can tell if we don't create that thread it succeeds without issue.

Right now my workaround is this.

    def __del__(self) -> None:
        loop = self._runner.get_loop()
        if sys.is_finalizing():
            default_executor = getattr(loop, "_default_executor", None)
            if default_executor is not None:
                default_executor.shutdown(wait=False)
                loop._default_executor = None
        self._runner.close()

I would love to personally not do this at all, I agree shutdown should be more ordered than relying on a __del__ method at module destruction time. But the code base using this is so complex and massive I can't just fix it, I'm working towards modernizing its usage of asyncio.

[kumaraditya303]

As far as I can tell if we don't create that thread it succeeds without issue.

That just fixes one instance of such issue, while interp is shutting down many modules are gone by the time finalizers run so several other APIs which rely on those modules globals would break in different ways so not all of them can be fixed easily.

[fried]

Alright if it stops working i'll just skip cleanup if finalizing. But should we have a warning in asyncio?

[kumaraditya303]

But should we have a warning in asyncio?

This isn't an asyncio issue, it happens with other modules as well so I don't think it should be added to asyncio docs explicity.

[ZeroIntensity]

__del__ is quite evil. You can break a lot by calling code in it, even moreso at finalization.