"RuntimeError: Event loop is closed" when ProactorEventLoop is used

[AndreyMZ]

Long story short

When ProactorEventLoop is used, the lack of graceful shutdown in aiohttp results in RuntimeError. (For comparison, when SelectorEventLoop is used, it results in ResourceWarning.) Note that ProactorEventLoop is the default event loop on Windows in Python starting from 3.8.

Maybe the root cause of this issue is the same as of #1925. But the impact is higher (RuntimeError instead of ResourceWarning).

Steps to reproduce

import asyncio
import sys

import aiohttp

async def main():
	async with aiohttp.ClientSession() as session:
		async with session.get('https://example.com/'):
			pass

if __name__ == '__main__':
	if sys.version_info[:2] == (3, 7):
		asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
	asyncio.run(main())

Actual behaviour

sys:1: ResourceWarning: unclosed <socket.socket fd=724, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('2001:0db8:85a3:0000:0000:8a2e:0370:7334', 19283, 0, 0), raddr=('2606:2800:220:1:248:1893:25c8:1946', 443, 0, 0)>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
C:\Programs\Python37\lib\asyncio\proactor_events.py:94: ResourceWarning: unclosed transport <_ProactorSocketTransport fd=-1 read=<_OverlappedFuture cancelled>>
  source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000017FAC405048>
Traceback (most recent call last):
  File "C:\Programs\Python37\lib\asyncio\proactor_events.py", line 95, in __del__
    self.close()
  File "C:\Programs\Python37\lib\asyncio\proactor_events.py", line 86, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Programs\Python37\lib\asyncio\base_events.py", line 683, in call_soon
    self._check_closed()
  File "C:\Programs\Python37\lib\asyncio\base_events.py", line 475, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Expected behaviour

No exceptions. Maybe warnings, but no exceptions.

Your environment

aiohttp (client) version: 3.6.2
Python version: 3.7, 3.8
OS: Windows 10

Additional information

I do not know any workaround.

A delay before closing the event loop suggested in the documentation does not help in a bit more complicated example (try to run it several times or add more URLs to the list):

import asyncio
import sys

import aiohttp

URLS = [
	# Insert several URLs here, e.g.:
	"https://www.python.org/ftp/python/",
	"https://github.com/python/",
	"https://www.cpan.org/src/5.0/",
	"https://aiohttp.readthedocs.io/",
	"https://example.com/",
]

async def main():
	async with aiohttp.ClientSession() as session:
		await asyncio.wait([fetch(session, url) for url in URLS])
	print("Done.")

async def fetch(session: aiohttp.ClientSession, url: str):
	async with session.get(url):
		print(f"Requested: {url}")

if __name__ == '__main__':
	if sys.version_info[:2] == (3, 7):
		asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
	loop = asyncio.get_event_loop()
	try:
		loop.run_until_complete(main())
		loop.run_until_complete(asyncio.sleep(2.0))
	finally:
		loop.close()

[asvetlov]

The aiohttp master will fix this. Sorry, the change cannot be backported to 3.x branch.

Big enough timeout may saturate the problem. Also, you can ignore all errors on the loop closing as an option.

That's sad that asyncio proactor raises an exception instead of an expected warning.
You know, asyncio on Windows is underpowered; neither @1st1 nor I am using asyncio on Windows for our daily job.
The fix of asyncio is very welcome, I'm happy to review the proactor's patch if any.

[HarsheetKakar]

import asyncio
import aiohttp

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def harsheet():
    async with aiohttp.ClientSession() as session:
        json = await fetch(session, "<databaseurl>")
        return (json)


async def pyrebase_test():
    async with aiohttp.ClientSession() as session:
        json = await fetch(session, "<databaseurl>")
        return (json)

from pprint import pprint

loop = asyncio.get_event_loop()

group1 = asyncio.gather(harsheet(), pyrebase_test())
results = loop.run_until_complete(group1)
pprint(results)
loop.close()

why is this code giving RuntimeError: Event loop is closed error?

[AndreyMZ]

This is not s solution, because SelectorEventLoop does not support some features which ProactorEventLoop supports. (See https://docs.python.org/3.8/library/asyncio-platforms.html#windows.)

[Pythonic-Rainbow]

Yes I did not read the topic clearly, I am sorry.

[thachnv92]

in
loop.run_until_complete(process())

[iann838]

Running asyncio.get_event_loop().run_until_complete(...) instead of asyncio.run(...) doesn't show the error. Is there any difference between them ?

EDIT: Peraphs there is because run_until_complete does not close the loop while asyncio.run does and calls _ProactorBasePipeTransport.__del__ where it raises the exception, I've provided a way to handle it in a dirty manner below.

[iann838]

EDIT (after almost 2 years of fighting with more and more aiohttp bugs)**

As another solution in another issue that is related and more to the root cause, use aiohttp 4.0 or use this code when closing client session #1925, monkey patching is not recommended if another solution exists.

Old monkey patching

I found another solution for this problem if some still having issues with it. This involves directly manipulating the class method itself, it might be dirty in some peoples eyes. This applies to situations where you need to close your loops after a tasks like web apps and other similar situations.

from functools import wraps

from asyncio.proactor_events import _ProactorBasePipeTransport

def silence_event_loop_closed(func):
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        try:
            return func(self, *args, **kwargs)
        except RuntimeError as e:
            if str(e) != 'Event loop is closed':
                raise
    return wrapper

_ProactorBasePipeTransport.__del__ = silence_event_loop_closed(_ProactorBasePipeTransport.__del__)

EDIT: You may want to do this only on Windows environment to avoid executing it in production (although ProactorEventLoop is not available on Linux, but just in case).

if platform.system() == 'Windows':
    # Silence the exception here.
    _ProactorBasePipeTransport.__del__ = silence_event_loop_closed(_ProactorBasePipeTransport.__del__)

[rbrasga]

@paaksing Thank you for that. Those errors were bugging me.

[TKperson]

@paaksing the code worked like gold. I have been scratching my hair off for days trying to fix this on my discord bot and then I saw this

[Aran-Fey]

I don't understand. Why is this still not fixed even though this issue has been closed for almost 1.5 years?

Can't you fix it? Can't you be arsed to fix it? What's going on?

[Aran-Fey]

@paaksing I see, thanks.

You're wrong about this being fixed in python 3.9 though. Guess we'll wait for python 3.10?

[iann838]

@Aran-Fey It's fixed for me on 3.9 tho, I honestly don't know the spaghetti behind this, but this is windows specific issue aswell. The workaround that I posted above should solve your issue.

[Dreamsorcerer]

Can't you fix it? Can't you be arsed to fix it? What's going on?

Also, refer to the comment that closed it and try not to be so rude: #4324 (comment)

[Aran-Fey]

@Dreamsorcerer I've seen that comment. It contains no useful information. "This will be fixed in master"? Clearly not. What am I supposed to take from that comment? There's nothing there except maybe the "neither of us uses asyncio on Windows" bit, which doesn't really mean anything either, but kinda sounds like "We don't know how to fix it".

Also, when was I rude?

[Dreamsorcerer]

@Dreamsorcerer I've seen that comment. It contains no useful information. "This will be fixed in master"? Clearly not. What am I supposed to take from that comment? There's nothing there except maybe the "neither of us uses asyncio on Windows" bit, which doesn't really mean anything either, but kinda sounds like "We don't know how to fix it".

Are you reproducing the error on master? If so, then perhaps the issue needs to be reopened, or a new issue made.
If not, the comment said the fix can't be backported to 3.x, meaning that it is probably not fixed in any public release currently.
If you think this is incorrect and the issue can be fixed in 3.x (I have no knowledge about this issue myself), then feel free to provide a fix in a PR.

Also, when was I rude?

In the comment I quoted, and your messaging still sounds aggressive...

[iann838]

@Aran-Fey
Let me ask you back: Why would the devs use it if:

1. It's not their issue.
2. It's not impacting functionality wise.
3. It's not affecting production environment.
   ?

Also I am not forcing you to include that in your project, you asked for a fix, I gave you a "fix that you said it is not", you still got the golden finger to decide what to do with your project.

[admirabilis]

The bug report is about ProactorEventLoop, this appears to only be available on Windows:
https://docs.python.org/3/library/asyncio-eventloop.html?highlight=proactor#asyncio.ProactorEventLoop

Which would suggest that you may be experiencing a different issue.

Ah, yes, I get a similar harmless RuntimeError, the message I'm ignoring is "Event loop stopped before Future completed."

[Dreamsorcerer]

The bug report is about ProactorEventLoop, this appears to only be available on Windows:
https://docs.python.org/3/library/asyncio-eventloop.html?highlight=proactor#asyncio.ProactorEventLoop
Which would suggest that you may be experiencing a different issue.

Ah, yes, I get a similar harmless RuntimeError, the message I'm ignoring is "Event loop stopped before Future completed."

Feel free to create a new issue with some code which reproduces it. It may just be something you overlooked in your code.

[NewUserHa]

see https://bugs.python.org/issue39232

[ttodua]

I don't think any of the above posted workarounds are ideal, either silencing the exception (not desirable), or setting SelectorEventLoopPolicy (which degrades functionality as mentioned above and here too).

Better workaround might be :

1. update python to 3.10.6, where the issue seems no longer happening.

or

2. to inject 0.25 sleep in shutdown function, like this, which avoids exception:

originalShutdownFunc = asyncio.BaseEventLoop.shutdown_default_executor
async def shutdown_default_executor(arg):
    result = await originalMet(arg)
    await asyncio.sleep(0.25)
    return result

asyncio.BaseEventLoop.originalShutdownFunc = shutdown_default_executor

asyncio.run(your_function) # HERE RUN YOUR ASYNCIO CODE

[iann838]

This is apparently fixed on aiohttp 4.0 beta. A more serious issue is involved here, a solid workaround was provided in this comment: #1925 (comment)
I will edit my previous comments to discourage the use of them.

[Mark-Jethro]

how to fix connectionreseterror: [winerror 10054] an existing connection was forcibly closed by the remote host

[NewUserHa]

asyncio.wait() before that loop closes, or use loop.run_until_complete instead of asyncio.run(). if remember correctly.

[waketzheng]

I don't think any of the above posted workarounds are ideal, either silencing the exception (not desirable), or setting SelectorEventLoopPolicy (which degrades functionality as mentioned above and here too).

Better workaround might be :

1. update python to 3.10.6, where the issue seems no longer happening.

or

2. to inject 0.25 sleep in shutdown function, like this, which avoids exception:

originalShutdownFunc = asyncio.BaseEventLoop.shutdown_default_executor
async def shutdown_default_executor(arg):
    result = await originalMet(arg)
    await asyncio.sleep(0.25)
    return result

asyncio.BaseEventLoop.originalShutdownFunc = shutdown_default_executor

asyncio.run(your_function) # HERE RUN YOUR ASYNCIO CODE

Worked for me with python3.11, but the code is as follows:

import asyncio

origin_shutdown = asyncio.BaseEventLoop.shutdown_default_executor
 
 
async def _shutdown(self):
    await origin_shutdown(self)
    await asyncio.sleep(5)
 
 
asyncio.BaseEventLoop.shutdown_default_executor = _shutdown  # type:ignore