Jupyter Notebok server process stays up after "shut down" due to `shutdown_no_activity_timeout`

[bensussman]

The jupyter process is staying up and not exiting even though it has "shut down" from shutdown_no_activity_timeout and is no longer listening on the configured port.

When configuring our Jupyter notebook (also run via Docker) we pass the following arguments:

jupyter notebook \
 --ip 0.0.0.0 \
 --allow-root \
 --no-browser \
 --log-level=DEBUG \
 --NotebookApp.allow_password_change=False \
 --NotebookApp.quit_button=False \
 --NotebookApp.shutdown_no_activity_timeout=60 \
 --MappingKernelManager.cull_idle_timeout=60 \
 --MappingKernelManager.cull_interval=10 \
 --MappingKernelManager.cull_connected=True \
 --MappingKernelManager.cull_busy=False

And the behavior we are seeing is that:

• It is correctly culling idle kernels after 60s
• It is (usually) correctly shutting down the server after there are no kernels and the no activity timeout has exceeded
• When the server is shutdown, the PROCESS DOES NOT EXIT so the docker container just hangs waiting for the process to exit
• Looking at netstat shows that our configured port 8888 was being LISTENED to before shutdown and is no longer being listened to post shutdown, but the container remains up (and therefore the process has not yet exited).
• When shutting down the server manually (not by the NotebookApp.shutdown_no_activity_timeout but by sending it a termination signal) it exits correctly.

One lead we have is that Websocket closed appears in the logs when shutting down the notebook manually, but does not appear when shut down due to shutdown_no_activity_timeout so our hypothesis is that there is a bug in the shutdown_no_activity_timeout code which is not correctly or fully shutting down the jupyter process.

This might be the same or similar issue as #5328 but unclear if that one is shutdown_no_activity_timeout related or if they are different. I didn't fully follow #4868 but it could also be related?

[kevin-bates]

Hi @bensussman - thanks for opening this issue.

I'm unable to reproduce this locally - are you able to reproduce this outside of docker?

It is (usually) correctly shutting down the server after there are no kernels and the no activity timeout has exceeded

Is this meant to imply the issue (i.e., docker stays running) is intermittent, or that you find the shutdown-no-activity functionality is not working at all - sometimes.

Are you using your own image or one produced via jupyter/docker-stacks?

[bensussman]

Hey @kevin-bates thanks for the reply! We are actually using a custom image (because we are providing a lot of other dependencies to our Jupyter users). You can see the image here: https://hub.docker.com/r/spellrun/default-cpu. I have not tried to reproduce it out of docker, but I will definitely give it a shot on Monday.

Good question re: intermittent. I am able to deterministically reproduce it in my current docker environment with a container running the default-cpu image. However when I spin up new environments (new EC2 instance, new container instance) I do see different behavior (including both not shutting down due to no-activity at all, and shutting down correctly and exiting the process) at first. One theory could be that some file-system-state within the docker container is causing the failure for Jupyter to cleanly shut down, and that's why brand new environments see different behavior? It's important to note that it's not a rare failure for us.

[kevin-bates]

I'm running the following and cannot seem to reproduce the issue in 3 attempts so far...

docker run -p 8888:8888 --rm -it  docker.io/spellrun/default-cpu:latest jupyter notebook  --ip 0.0.0.0  --allow-root  --no-browser  --log-level=DEBUG  --NotebookApp.allow_password_change=False  --NotebookApp.quit_button=False  --NotebookApp.shutdown_no_activity_timeout=60  --MappingKernelManager.cull_idle_timeout=60  --MappingKernelManager.cull_interval=10  --MappingKernelManager.cull_connected=True  --MappingKernelManager.cull_busy=False

Can you please sure the command you use to reproduce the issue - as well as actions you take once the container is running? In the 3 attempts, 2 were simply startup and let expire and the 3rd started a kernel, which I let be culled, then the server shutdown in the appropriate poll interval after that.

Thanks.

[bensussman]

Thanks @kevin-bates. I wanted to give you a quick update on what i've been doing: I have been banging away at this all day and am unable to reproduce it using a docker run command. This must mean that the docker container context created by our golang api is causing this bug. When I do a docker inspect of the container created by docker run vs the container created via our golang api I see quite a few differences (things like NetworkMode, Dns, Tmpfs, etc.) and my only remaining theory is that one of those unique values for the container is causing this issue. I have eliminated disk state as being a cause, as I am able to completely remove any source of disk state and still reproduce the error. I have also eliminated tornado_settings and custom_display_url as being causes, as I am able to pass them to docker run and NOT display the error. I would ask that you ignore this issue for now, and I will reopen it (hopefully very soon) when I have identified the context that triggers this behavior and thus have a better reproducibility story for you. Thanks again for your time!

[kevin-bates]

Thanks for the great update Ben - this is much appreciated! Yes, should you find the solution please post back here, even it doesn't warrant the re-opening of this issue. That way others may be able to learn from your solution.

[rschroll]

I think I may be seeing this same issue. The environment is a bit different -- we're running JupyterHub on k8s -- but the behavior is very similar. Here's what I've noticed:

• If there is no browser window with a notebook open, the server shuts down correctly. This happens even if there is a running kernel.
• If there is a browser window with a notebook open, the server seems to hang on shutdown.
• It doesn't matter what kernel the notebook is using (at least of the two I tested).
• If you manually shut down the notebook kernel from the "Running" tab, but leave the notebook open, it will still hang. (It doesn't seem to be related to shutting down the kernel.)
• If you manually shut down the notebook kernel from the notebook interface, but leave the notebook open, it will shut down correctly (!).
• Activities that try to contact the singleuser server result in 503 errors, but the JupyterHub interface still shows the server up. Shutting it down and starting it up restores functionality.
• If I get a shell in the container running the singleuser server after it hangs, I can see that the process is still running. However, it is not responding on port 8888. (This is why we're getting 503s, I imagine.)

Adding in @bensussman's discovery about the websockets, my bet is that something in the server is waiting for a websocket connection to the notebook to be closed, and this is keeping the server process active. Closing the browser tab, or shutting down the kernel from the notebook, hangs up the websocket correctly, so we don't get the hang.

We're using a custom singleuser image. I can probably try with a standard image, if you think that would be useful. Just let me know which to try. Versions of things that might be relevant:

jupyter                   1.0.0
jupyter_client            6.1.0 
jupyter_console           6.1.0 
jupyter_core              4.6.3
jupyterhub                1.1.0
ipykernel                 5.1.4
ipython                   7.12.0
notebook                  6.0.3

Let me know if there's any additional debugging that would help.

[rschroll]

I can reproduce the problem using the jupyterhub/k8s-singleuser-sample image, version 0.9.1.

[rschroll]

I've managed to reproduce the problem with a docker container running locally. I launch a Jupyter server with the following command (adapted from above):

docker run -p 8888:8888 --rm -it  docker.io/jupyterhub/k8s-singleuser-sample:0.9.1 jupyter notebook  --ip 0.0.0.0  --allow-root  --no-browser  --log-level=DEBUG  --NotebookApp.allow_password_change=False  --NotebookApp.quit_button=False  --NotebookApp.shutdown_no_activity_timeout=60  --MappingKernelManager.cull_idle_timeout=60  --MappingKernelManager.cull_interval=10  --MappingKernelManager.cull_connected=True  --MappingKernelManager.cull_busy=False

Once again, I notice it hang when there is a browser tab left open for a notebook. Specifically, when the server tries to shut down, I see:

• Success - Open nothing in the browser
• Success - Open only a directory view
• Hang - Open a notebook, let the kernel culler stop the kernel
• Hang - Open a notebook, kill the kernel from the "Running" tab
• Success - Open a notebook, shut down the kernel from the notebook
• Success - Open a notebook, close the notebook, and then kill the kernel from the "Running" tab
• Success - Open a notebook, close the notebook, and let the kernel culler stop the kernel

This is running a recent docker image, so I suspect it's still a problem. Can we get this issue reopened?

[kevin-bates]

Thanks for the great troubleshooting @rschroll! I agree this warrants deeper investigation and likely a fix, but we're probably going to need to eliminate the container aspect to get to the bottom of things. I'm away for the rest of the week and probably can't look into any time soon anyway, so I'm hoping this will spark someone's interest.

Sounds like it requires a connected kernel that has been culled or shutdown outside of the notebook itself.

Thanks again.

[rschroll]

Thanks for reopening! I've managed to reproduce the hang locally. More excitingly, I can also tweak it to not hang. This environment seems to reliably hang in the conditions discussed above (notebook open, kernel not shut down by notebook). However, it is sensitive to version numbers. If I upgrade notebook to 6.1.3 (from 6.0.3) OR jupyter-client to 6.1.6 (from 6.1.0), I do not get the hang. That is, it is the interaction of these two versions that cause the hang.

I also noticed a difference in the debug output in these cases. When the kernel is culled when the system halts correctly, I see this output:

[W 20:08:21.343 NotebookApp] Culling 'idle' kernel 'python3' (dc8f9a3a-68cf-4332-b51b-c723202b1c68) with 1 connections due to 68 seconds of inactivity.
[D 20:08:21.344 NotebookApp] Clearing buffer for dc8f9a3a-68cf-4332-b51b-c723202b1c68
[I 20:08:21.344 NotebookApp] Kernel shutdown: dc8f9a3a-68cf-4332-b51b-c723202b1c68
[D 20:08:21.765 NotebookApp] Websocket closed dc8f9a3a-68cf-4332-b51b-c723202b1c68:4b7b26b5dd7647d88b05acf743b26b0f
[W 20:08:22.774 NotebookApp] 410 DELETE /api/sessions/4c318e03-2833-450f-9d30-6dc639c85c52 (127.0.0.1): Kernel deleted before session
[W 20:08:22.776 NotebookApp] Kernel deleted before session
[W 20:08:22.778 NotebookApp] 410 DELETE /api/sessions/4c318e03-2833-450f-9d30-6dc639c85c52 (127.0.0.1) 8.87ms referer=http://127.0.0.1:8888/notebooks/ipy-scratch/Untitled7.ipynb

However, when the system does not halt correctly, I only see the first three lines. I don't see the websocket closing or the session being deleted. This would fit with the hypothesis that on open websocket is keeping the server from exiting.

So it may be that I just got an unlucky set of versions in the image I was using. I'll build a new image with updated versions and see if that works on k8s as well.

[kevin-bates]

Excellent information Robert - thank you. NB 6.1 does include some refactoring of kernel management to add support for async kernel management and jupyter_client 6.1.6 (actually 6.1.5) includes a fix to properly close the ZMQ context that may be coming into play here. What I don't really understand is why its NB 6.1.3 OR JC 6.1.6. However, if both are in play, then simply a new image set might be all that is necessary.

I'm taking the rest of the week off, so probably won't be able to move anything forward, but it sounds like you're in good shape anyway and we may be able to address this issue by documenting the update or actually bumping the JC dependency.

Thanks again for the great work!

[rschroll]

I've built and deployed a new image with the updated versions of both notebook and jupyter-client. This solves the problem I had noticed on JupyterHub. I'm happy to help debug further, if it would be useful. (Although I'll note that I'll be gone next week.) But I'm also happy to mark this as fixed and close the issue.

[kevin-bates]

I definitely believe there's a tie to the closing of the ZMQ context change in jupyter_client 6.1.5+ since as soon as I install 6.1.5 (running with NB 6.0.3), the hang does not occur. However, I'm unable to reproduce this using NB 6.1.0 and JC < 6.1.5, but wouldn't be surprised if the async kernel management work has something to do with what you're seeing.

At any rate, it looks like the jupyter stacks base notebook image was updated from 6.0.3 to 6.1.3 4 days ago, so I suspect we're good on that front as well and believe the best course of action at this point is to close this.

Thank you @rschroll and @bensussman for all the help in identifying and troubleshooting this!