RuntimeError: "This event loop is already running" with nbclient==0.8.0

[taras]

Related #330 but happening with nbclient==0.8.0

Description

When I execute a job, I get "This event loop is already running" error in the UI.

In the service logs, I get the following error.

Sep 29 11:57:51 tubuntu jupyter-notebook[857]: [I 2023-09-29 11:57:51.712 ServerApp] Connecting to kernel a687ffa6-7465-4686-b209-7c1af177a8c2.
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]: Traceback (most recent call last):
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/jupyter_scheduler/executors.py", line 60, in process
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     self.execute()
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/jupyter_scheduler/executors.py", line 140, in execute
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     ep.preprocess(nb)
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/nbconvert/preprocessors/execute.py", line 94, in preprocess
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     with self.setup_kernel():
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/contextlib.py", line 137, in __enter__
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     return next(self.gen)
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:            ^^^^^^^^^^^^^^
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/nbclient/client.py", line 596, in setup_kernel
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     self.start_new_kernel(**kwargs)
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/jupyter_core/utils/__init__.py", line 168, in wrapped
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     return loop.run_until_complete(inner)
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/asyncio/base_events.py", line 629, in run_until_complete
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     self._check_running()
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/asyncio/base_events.py", line 588, in _check_running
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     raise RuntimeError('This event loop is already running')
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]: RuntimeError: This event loop is already running
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]: /home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/jupyter_scheduler/executors.py:64: RuntimeWarning: coroutine 'NotebookClient.async_start_new_kernel' was never awaited
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   self.on_failure(e)
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]: RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Sep 29 11:58:22 tubuntu jupyter-notebook[857]: [W 2023-09-29 11:58:22.103 ServerApp] Notebook Matrix optimization/Post Sep 29/scripts/Trust_ColAdd_1.ipynb is not trusted

Reproduce

I'm using mamba to install Jupyter Notebook & JupyterLab and PIP to install Jupyter scheduler.

Expected behavior

Would love to be able to run jobs :(

Context

• Troubleshoot Output
• Command Line Output

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[kinaumov]

Have the exact same issue. Have tried a variety of things already, still getting the same response.

[samuelan]

try Downgrade jupyter_core to 5.3.1

[kinaumov]

Thank you very much sir @samuelan, resolved.

[taras]

It made this problem go away for me too. Thank you 🙏

Do we close this issue or leave it open since it's still a problem with 5.3.2?

[samuelan]

I found the issue with jupyter_core==5.3.2 with a different problem that I had. I haven't figured out the exact steps to reproduce the issue without sharing internal codes. If you folks have a simplified reproduction scenario you can share here, that'd be great. I commented on one PR in jupyter_core which change the way event loop was handled in run_sync function. The author of that PR was asking my repro, and so far I am not able to give him that.

[taras]

I'm guessing that reproducing my environment would probably do it. I could probably do it with some kind of sandbox that supports python.

[gbellvelocity]

I am having a similar issue, but I am installing most packages with conda install and my jupyter_core=5.4.0 and nbclient=0.8.0 and I am running into similar errors. I have tried downgrading to jupyter_core=5.3.1 and still get the same error. I am running user instances via jupyterhub-singleuser. From other posts, I have also tried to set nbclient=0.7.1 and issue still persists. I have seen other posts about asyncio python module, but they were older so assuming this is not the issue.

[samuelan]

For me, the combination that worked was
jupyter_core==5.3.1
nbclient==0.8.0
nbclassic==1.0.0

I have seen nbclient==0.5.13 and nbclassic==0.4.7 being not compatible/broken with jupyter_core==5.3.1. So jupyter_core==5.3.1 by itself is not a silver bullet. The PR that change the event loop was in jupyter/jupyter_core#362, and I made a comment there about that. I have not been able to reproduce the same problem with simple Python codes on my laptop, but once I do, I will post here and over there at jupyter_core about it.

[gbellvelocity]

@samuelan Thanks for the prompt response. I did not have any luck with the settings above. Before I started any version modifications I was at:

jupyter_core==5.4.0
nbclassic==0.5.5
nbclient==0.8.0

I am actually running on Amazon2 kernel if that helps. I am not using containers either, long story as to why not, but customer is always right. :)

I typically create a base environment, and then I run "conda list --explicit > user.txt" and build/create user envs from that.
conda --version is 23.9.0

@samuelan Sorry for being an idiot. The settings you provided above worked when I dropped them into the base environment as I run "jupyterhub" process is ec2-user from which my pre_spawn_script creates a user env if it does not exist. So finally paid closer attention to the logs and realized that it was failing in base jupyterhub. Sorry about that. I will keep an eye out for any additional updates. I left my comments above so that you had the full story.

[samuelan]

@gbellvelocity no worries. Glad it worked for ya.
I still don't have simple repro yet for the issue on my laptop.

[dlqqq]

@samuelan @taras Hey folks, I just ran into this issue myself. Sorry for the lack of communication from us maintainers on this issue, we've been super busy as of late 😓. First, I want to document a few things explicitly:

1. I am able to reproduce this bug with jupyter_core==5.4.0.
2. Downgrading to jupyter_core==5.3.1 as recommended by @samuelan worked for me.

I see that @samuelan went the extra mile and did some great investigatory work, tracing the bug to jupyter/jupyter_core#362. I have a pretty solid understanding of async and Jupyter, so I think I can fix this bug directly on jupyter_core. If it can't be done in a timely manner, I'll cut a patch release to force a version constraint like jupyter_core>=5,<=5.3.1 to quickly fix this bug for existing users.

Thank you all for using, contributing, and debugging Jupyter Scheduler! Love the support.

[samuelan]

Thanks a lot, @dlqqq. I tried multiple rounds of reproducing the problem with simpler setup in vain. The codes that crashed for me was in executors.py, where I customized it and used a
run_sync(kernel_manager.start_kernel) call.

the PR362 in jupyter_core does not create new event loop each time, but try to reuse existing running loop. but I could not figure out what caused the sanity check in asyncio.base_events to fail.

https://github.com/python/cpython/blob/43a6e4fa4934fcc0cbd83f7f3dc1b23a5f79f24b/Lib/asyncio/base_events.py#L581-L583

supposedly after a new event loop is created for the current process/thread, it's not supposed to be running till run_until_complete or run_forever is called.

In any case, it sounds like you got a good handle on this. I will stay tuned to your fix. Thanks a lot, @dlqqq!

[dlqqq]

Hey folks! Good news. I found the root cause of the bug. It turns out that this bug has an oddly specific set of conditions to manifest:

• POSIX environment (i.e. Linux)
• jupyter_core>=5.3.2 (at the time of writing)
• Python <3.12

I'm closing the issue for now as it will be fixed in the next release, which will occur in the next 24 hours.

@samuelan You might be interested in the technical details of this bug. It's a three-way interaction between multiprocessing, asyncio, and the Linux kernel 😁. It's all documented in the description of #450.

[samuelan]

@dlqqq, thank you so much! No wonder I could not reproduce it locally. So it had to be on Linux, while I was on a Mac trying to reproduce. Right ahead of Halloween, this is rather wicked bug. Really appreciate solving it for us.

[guigoruiz1]

Hello,

I am currently having the exact same error with the conditions as @dlqqq mentioned:

POSIX environment (i.e. Linux)
jupyter_core>=5.3.2 (at the time of writing)
Python <3.12

[dlqqq]

@guigoruiz1 This is fixed in jupyter_scheduler>=2.3.0 and jupyter-scheduler>=1.6.0.