asyncio.exceptions.CancelledError

[CatamountJack]

Question

With NiceGUI 1.4.26 on Python 3.11.9 (x64) on Windows 11, I'm often getting an asyncio.exceptions.CancelledError when I exit my application. The exception doesn't show up in the console anywhere, only in the log file and always at the very end.

I'm not sure if it's because I'm misusing async somehow (I'm still fairly new to it) or if something else is going on. I've set an app.on_exception() handler on the application, but it's not being propagated through there. I've also wrapped my .run() call in a context manager to suppress asyncio.exceptions.CancelledError, but it still shows up.

In case any of this is relevant:

• I'm exiting using an exit handler which calls app.native.main_window.destroy() and app.shutdown()
• The app is not frozen, I'm running it in PyCharm, and I've set my PyCharm console to Emulate terminal output in console.
• The application is using some threads which I'm running with await loop.run_in_executor(...) and one of those calls does happen from another function that's called at the beginning of my exit handler (which is to save application data before exiting)

I saw another discussion here with the same title asyncio.exceptions.CancelledError, but that was for a PyInstaller-bundled application and they were using Ctrl-C to exit, so I'm posting this separately.

I don't have a minimal reproducible example at this point, because I'm not sure where in my code the error is originating from.

Here's the traceback, as it appears in my log file:

2024-06-22 09:26:42 ERROR     base_events                    default_exception_handler      1785  Exception in callback page.__call__.<locals>.check_for_late_return_value(<Task cancell...\page.py:110>>) at C:\dev\source\project\.venv\Lib\site-packages\nicegui\page.py:96
handle: <Handle page.__call__.<locals>.check_for_late_return_value(<Task cancell...\page.py:110>>) at C:\dev\source\project\.venv\Lib\site-packages\nicegui\page.py:96>
Traceback (most recent call last):
  File "C:\Users\catamountjack\.rye\py\cpython@3.11.9\Lib\asyncio\events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "C:\dev\source\project\.venv\Lib\site-packages\nicegui\page.py", line 97, in check_for_late_return_value
    if task.result() is not None:
       ^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

Any ideas?

[CatamountJack]

I removed the calls to await loop.run_in_executor(...) and directly awaited the function (that saves my application data at the start of my exit handler), but I'm still having the issue, so it doesn't appear to be related to the additional threads.

[falkoschindler]

Hi @CatamountJack,

In some of our projects we also get occasional CancelledErrors, but we haven't narrowed it down to the root cause. We guess the problem lies within user code, i.e. not the NiceGUI library itself.

Does anyone else get these errors?

[CatamountJack]

Thanks @falkoschindler

I'm WAY outside my area of expertise here, so please forgive me if this is a dumb suggestion/question.... but would it make sense to catch and suppress CancelledError exceptions in page.page._call__.check_for _late_return_value()? Based on the error being logged, it looks like, by that point, any tasks should have returned anyway. My (very limited) understanding is that CancelledError isn't really an error, but provides a way of signaling that the task has been cancelled while allowing some task cleanup to happen... is there other task cleanup that would be skipped if it were suppressed at that point?

[falkoschindler]

Your comment made me think once again and I actually found a reproduction:

import asyncio
import logging
from nicegui import app, ui

logging.basicConfig(level=logging.INFO, filename='test.log')

async def task():
    try:
        while True:
            await asyncio.sleep(1)
            logging.info('Tick')
    except:
        logging.exception('Error')

app.on_startup(task)

ui.button('Shutdown', on_click=app.shutdown)

ui.run(reload=False)

What is happening here?

1. We want to catch any exceptions during an async background task and use a try-except block. That's ok.
2. But instead of writing except Exception: which would catch all subclasses of Exception, we write except: which catches all subclasses of BaseException which includes asyncio.CancelledError as well as KeyboardInterrupt (see asyncio.exceptions.CancelledError #1153). This is a common error and code checkers like Ruff or Pylint warn about it.
3. Once we call app.shutdown (or press Ctrl-C), the exception is raised, caught and logged.

Of course this example doesn't rule out any bug in NiceGUI causing a similar result. But there's reason to assume that the (base-)exception is accidently caught and logged in user code.

[CatamountJack]

Nice investigative work! :-) I just searched through my code base and I don't have any bare exceptions. I found that there are a lot of bare exceptions in other packages though... is it possible it's being caught in one of them and propagated up through?

[falkoschindler]

I'm not sure. Since you seem to be able to reproduce the problem consistently, maybe you can reduce your app step by step and see when the logged exception disappears. This would help us to narrow it down. Or does it also occur when running a simple Hello World app?

[CatamountJack]

I don't know if this is the only thing that causes it because I think I get it elsewhere where I'm not awaiting a button click (still investigating that), but I can reproduce it like this...

from nicegui import ui

@ui.page('/')
async def main():
    await ui.button("Button").clicked()

if __name__ in {"__main__", "__mp_main__"}:
    ui.run(native=True, reload=False, )

The error shows up if you close the window while awaiting the button click.

Which kinda makes me to blame, since I was the one that requested awaitable button clicks in the first place. My apologies!

[falkoschindler]

Very interesting! That's a whole new vector. It seems like awaiting anything within page functions causes CancelledErrors when the app terminates meanwhile:

@ui.page('/')
async def page():
    await ui.context.client.connected()
    await asyncio.sleep(10)

Killing this app a few seconds after visiting the page reproduces the problem.

Which kinda makes me to blame

Absolutely not! Every new feature comes with some risk of containing a bug or two. We all missed this one, but found it eventually. And it just unveils a much more general issue somewhere in page.py, I guess.

[falkoschindler]

The problem seems to be caused by the check_for_late_return_value callback we introduced 7 months ago. When removing it, the CancelledError is gone.

So I boiled it down to this:

def bug():
    task = asyncio.create_task(asyncio.sleep(10))
    task.add_done_callback(lambda t: print(t.result()))

app.on_startup(bug)

Next step: Find out why we can't use asyncio this way.

[falkoschindler]

I think I fixed it in PR #3296.

[CatamountJack]

Which kinda makes me to blame

Absolutely not! Every new feature comes with some risk of containing a bug or two. We all missed this one, but found it eventually. And it just unveils a much more general issue somewhere in page.py, I guess.

I meant that a little tongue-in-cheek (it's hard to convey intent in writing sometimes), but you're very kind, thanks!

I monkey patched my version with your fix and it seems to take care of the issue. Thanks for the quick fix - I'm always impressed at how responsive you guys are!!