ctrl-c / KeyboardInterrupt makes event loop exceptions

[mguijarr]

Exiting asyncio.run(<a main function()>) with SIGINT (ctrl-c, KeyboardInterrupt exception) makes bad RuntimeError exceptions. I am affected by this in a prompt-toolkit application relying on gevent.

The minimum code to reproduce the problem:

import gevent
import gevent.monkey; gevent.monkey.patch_all()
import asyncio
import asyncio_gevent
import time

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

async def my_main():
    gevent.sleep(10)

# press CTRL-C to exit the Python program before 10 seconds
asyncio.run(my_main())

python test.py
^CKeyboardInterrupt
2022-11-16T14:00:43Z
Traceback (most recent call last):
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/base_events.py", line 574, in run_until_complete
    self.run_forever()
  File "/home/matias/dev/asyncio-gevent/asyncio_gevent/event_loop.py", line 19, in run_forever
    greenlet.join()
  File "src/gevent/greenlet.py", line 833, in gevent._gevent_cgreenlet.Greenlet.join
  File "src/gevent/greenlet.py", line 859, in gevent._gevent_cgreenlet.Greenlet.join
  File "src/gevent/greenlet.py", line 848, in gevent._gevent_cgreenlet.Greenlet.join
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/runners.py", line 46, in run
    _cancel_all_tasks(loop)
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/runners.py", line 62, in _cancel_all_tasks
    tasks.gather(*to_cancel, loop=loop, return_exceptions=True))
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/base_events.py", line 563, in run_until_complete
    self._check_runnung()
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/base_events.py", line 523, in _check_runnung
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/bla.py", line 12, in <module>
    asyncio.run(my_main())
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/runners.py", line 50, in run
    loop.close()
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/unix_events.py", line 55, in close
    super().close()
  File "/home/matias/miniconda3/envs/bliss/lib/python3.7/asyncio/selector_events.py", line 88, in close
    raise RuntimeError("Cannot close a running event loop")
RuntimeError: Cannot close a running event loop

[gfmio]

Thanks for reporting, I'll look into this!

[gfmio]

I haven't found a complete solution yet, but the problem appears to be that gevent.sleep pauses the current greenlet, so that nothing in it will execute.

[gfmio]

I've found the reason for this behaviour and implemented a fix in #10. The change has been released as version 0.2.3.

The issue is that the gevent.sleep is causing the running greenlet to pause, which pauses the asyncio event loop by default.

The solution is to run any code that runs gevent.sleep in another greenlet using asyncio.sync_to_async or (equivalently) asyncio_gevent.greenlet_to_future(gevent.spawn(f)). This causes the function to be executed in another greenlet which prevents the main greenlet from getting blocked.

Example:

import gevent.monkey

gevent.monkey.patch_all()

import asyncio
import threading

import gevent

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())


async def f():
    print("f", 1)
    await asyncio.sleep(1)
    print("f", 2)


def g():
    print("g", 1)
    gevent.sleep(2)
    print("g", 2)


async def main():
    await asyncio.gather(f(), asyncio_gevent.sync_to_async(g)())
    # OR equivalently
    # await asyncio.gather(f(), asyncio_gevent.greenlet_to_future(gevent.spawn(g)))


if __name__ == "__main__":
    asyncio.run(main())

The output will be (as expected):

g 1
f 1
f 2
g 2

If gevent.sleep is called inside an async function, then the async function needs to first be wrapped in asyncio.async_to_sync.

import gevent.monkey

gevent.monkey.patch_all()

import asyncio

import gevent

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())


async def f():
    print("f", 1)
    await asyncio.sleep(1)
    print("f", 2)


async def g():
    print("g", 1)
    await asyncio.sleep(1)
    gevent.sleep(1)
    print("g", 2)


async def main():
    await asyncio.gather(
        f(), asyncio_gevent.sync_to_async(asyncio_gevent.async_to_sync(g))()
    )


if __name__ == "__main__":
    asyncio.run(main())

The output will again be (as expected):

g 1
f 1
f 2
g 2

In both cases, you can Ctrl+C anytime and you will receive the standard asyncio behaviour of triggering a CancelledError.

I hope this fixes your issue as well. Please re-open the ticket if you still encounter this issue.

[spumer]

I have a suggestion to optmize this workaround.

Right now we use default executor (None) to join greenlet, and this can produce a lot of threads or can get deadlock if no more threads can be spawned to join greenlet (context switch)

wait_executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)

async def _await_greenlet(
    ...      
        result, _ = await asyncio.gather(
            future, loop.run_in_executor(wait_executor, gevent.sleep, 0)  # or we can use gevent.idle
        )

I was testing it with max_workers=1 and greenlet.join and got stuck on loading django admin. gevent.sleep / gevent.idle allow context switching and instantly free thread. As result no deadlocks and threads overhead and clear traceback at CTR+C

[spumer]

Can i make a PR? :)

[gfmio]

Sure, go ahead :)