Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No warning/error message when kernal is forced to restart #4273

Closed
NicWayand opened this issue Mar 29, 2018 · 12 comments
Closed

No warning/error message when kernal is forced to restart #4273

NicWayand opened this issue Mar 29, 2018 · 12 comments

Comments

@NicWayand
Copy link

@NicWayand NicWayand commented Mar 29, 2018

Sometimes when running a notebook the kernal restarts (in my case due to a OOM issue) but does not give the user any message. The cell will still show an asterisk, as if its running, but only when the user hovers the mouse over the circle at the top right, will it show that the kernal is restarting.

I would expect a larger warning to the user, either in the current cell, or at the top of the notebook, to let them now some issue has occurred.

In the terminal where jupyter lab was started gives:
KernelRestarter: restarting kernel (1/5), keep random ports

jupyterlab 0.31.12

@clkao
Copy link

@clkao clkao commented Oct 22, 2018

This happens in 0.34.7 as well.

@clkao
Copy link

@clkao clkao commented Dec 24, 2018

Note for plain old notebook, there's a simple popup for OOM/restarted kernel, which should be informative enough.

2018-12-25 12 17 05

@kykrueger
Copy link

@kykrueger kykrueger commented Feb 7, 2019

I'd like to see more details as to why the kernel has had to restart. Some sort of stack trace from what the kernel was trying to execute.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Feb 7, 2019

That would be great, however if it was an OOM, it's killed suddenly without a stack trace, IIRC.

@kykrueger
Copy link

@kykrueger kykrueger commented Feb 7, 2019

How can we tell if it was an OOM problem?

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Feb 8, 2019

With our current infrastructure, I'm not sure you could easily. The process is gone and we just notice it no longer is responding to heartbeats.

@kykrueger
Copy link

@kykrueger kykrueger commented Feb 8, 2019

Gotcha,

Some of the jupyter docs for the architecture were really helpful for understanding where the limitation lies. Would I be correct to assume that this crash and restart is caused by a timeout between the kernel and notebook server?

Is this something that the notebook server has an interface for, but the kernel has not implemented?

I'll try and get some errors out of my installed version of ipykernel later.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Feb 8, 2019

Is this something that the notebook server has an interface for, but the kernel has not implemented?

Right. There is a message the server sends the frontend that indicates the restart happened without a user request, and JupyterLab currently is not distinguishing between that and a restart requested by the user. The classic notebook notifies the user if a restart happens that was not requested.

I worked on this and many other things in #4724, but that turned into a much more ambitious project that should be postponed until after 1.0. I still plan to look again at this specific issue to investigate solving just it (#4748 is the 1.0 issue for this, though perhaps this should be the 1.0 issue and that should be closed as a duplicate).

@jasongrout jasongrout removed this from the Future milestone Feb 8, 2019
@jasongrout jasongrout added this to the 1.0 milestone Feb 8, 2019
@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Feb 8, 2019

I still plan to look again at this specific issue to investigate solving just it

And as with any issues, help is welcome! If anyone wants to take this up, please comment on the issue so that work isn't duplicated.

@kykrueger
Copy link

@kykrueger kykrueger commented Feb 10, 2019

I worked on this and many other things in #4724, but that turned into a much more ambitious project that should be postponed until after 1.0.

@jasongrout that looks like quite the rabbit-hole. I'd be quite happy if we can solve this separately. I've never used TypeScript or JavaScript, but I'll see if I can help.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Apr 24, 2019

It seems that killing the kernel process so that it autostarts does not work in jlab (I can't use the newly started kernel)

@lock lock bot locked as resolved and limited conversation to collaborators Aug 7, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

5 participants