Output do not update after close and reopen the notebook page

[oeway]

Hi,

I am using a library named keras with jupyter notebook. It's nice that keras use a text progress bar to indicate the progress like this:

Train on 381734 samples, validate on 20092 samples
Epoch 1/1
381734/381734 [==============================] - 257s - loss: 0.4094 - val_loss: 0.2863

But the problem is that, if the script is running with the progress bar, I can't close the webpage which I used to start the script. Otherwise I won't see the progress bar update anymore, when I open the notebook again, if it's still running, the title will become something like "(Starting) XXXXXX", but I can't see any update in the page.

The progress bar used by keras is like this:
https://github.com/fchollet/keras/blob/b126b6328a44fa3332d2d7fd011da3ff196a669a/keras/utils/generic_utils.py

Can anyone look into this?

Thanks in advance.

[willingc]

Closing this issue as it's been inactive for several months. Please feel free to open a new issue at the jupyter/help repo or reopen this issue. Thanks! 🌻

[hadim]

I am facing the same issue. Is it possible to reopen a notebook and get the output of the running kernel ?

[raviqqe]

I have the same issue too. Reconnecting kernel doesn't help me either.

[willingc]

@raviqqe Does this happen with Keras or any notebook?

[raviqqe]

I was running not Keras but numpy code. I closed a notebook tab and opened it again. However, it's not showing updated output while its kernel is still running. Note that I was running Jupyter in client-server setting.

My environment is the below.

• Jupyter: 4.2.0
• Browser: Google Chrome 53.0.2785.154
• Host OS: Ubuntu 16.04.1 LTS (Xenial Xerus)
• Client OS: Chrome OS

The early part of my configuration file ~/.jupyter/jupyter_notebook_config.py:

# Configuration file for jupyter-notebook.

#------------------------------------------------------------------------------
# Application(SingletonConfigurable) configuration
#------------------------------------------------------------------------------

## This is an application.

c = get_config()
c.NotebookApp.open_browser = False
c.NotebookApp.password = 'sha1:6cea77f:673802e58d129779298e501a2bc40'
c.NotebookApp.certfile = '/home/raviqqe/mycert.pem'
c.NotebookApp.keyfile = '/home/raviqqe/mykey.key'
c.NotebookApp.ip = '*'
c.NotebookApp.port = 8888

## The date format used by logging formatters for %(asctime)s
#c.Application.log_datefmt = '%Y-%m-%d %H:%M:%S'
...

I'm not sure that this problem is caused by Jupyter or Chrome.

[sorenwacker]

Same problem

[floatdrop]

@willingc can you reopen this issue? It hit me as well.

[willingc]

@floatdrop Instead of reopening here, would you mind opening a new issue on jupyter/notebook repo? Also Jupyter notebook has a new (2 days ago) 5.0 release. Perhaps upgrade and see if the issue is still present. Thanks!

[tadas-subonis]

The issue is still here

[ajinkyaT]

Same problem

[gagolucasm]

Same issue

[willingc]

@gnestor Would you mind taking a look and see if you can reproduce? If you can, would you mind moving to this issue to the notebook repo? Thanks.

[gnestor]

My guess: When a notebook is closed and reopened, the kernel may continue running but the cells' parent header values change. Therefore the notebook reconnects to the kernel and resumes receiving messages from the kernel but the cell outputs don't update because they don't recognize the parent headers in these messages.

@rgbkrk Do you concur?

The simple test case:

import time

for i in range(101):
    time.sleep(1)
    print(i)

Then reload the notebook and if you're logging messages, you'll see that the stream messages continue to come in but they're not being appended to any cell's output.

What I observe:

• The cell ids change after reloading or closing/reopening (e.g. Jupyter.notebook.get_cells())
• The msg.parent_header.msg_id is consistent for stream messages originating from the same cell, so even after the notebook is reloaded or closed/reopened, the parent_id is the same

I'm not sure where in the source messages are mapped to cells outputs (and specifically where the parent_id is used to do that), but I assume it has something to do with the cell's id which changes after a reload or close/reopen.

[rgbkrk]

Yup, there's no more association between the cells and the message IDs when the page is refreshed because that's all stored client side in the browser.

[cahya-wirawan]

I have also the same problem, it's annoying if we can't see the progress anymore after we close the tab or disconnected for a while, especially during the training which could run for many hours or days. So, fix to this bug would be really very helpful. Thanks.

[bbennett36]

please.
add.
this.
feature.

My app has been running for hours and I can't tell how far it is :/

[jrubinstein]

please please please fix this issue! I end up having to re-run code because the output doesn't update if I have to disconnect. This is a huge problem for scripts that take a long time ... which they do.
Using R, Chrome, on Mac

[Xpitfire]

Any update on this issue? It is impractical to operate this way on remote platforms without output updates after closing a notebook. For example Google Cloud engines runs Jupyter notebooks and the connection gets lost from time to time, but the kernel continues operating and without progress updates.

[gnestor]

@rgbkrk @minrk @ellisonbg What changes would be needed to preserve the mapping between message parent header IDs and cell IDs? Could this somehow be derived from the notebook document? Or could it be stored in local storage?

[rgbkrk]

At least for nteract and jupyterlab, that seems like a reasonable thing to store in the localStorage since they both have cell IDs.

[mehditlili]

I didn't have this problem before (feb 2019) but now I have it somehow. Training for 2 days and no way to see the current status..

[tausif-0311]

Any solution for this?

[gnestor]

Unfortunately no, the cell IDs change whenever the page is refreshed so the mapping between kernel messages and cells is lost. I don't have time to work on this at the moment but I think it's an important feature and I will support anyone that wants to take it on.

[dineshdaultani]

Same issue. It's really frustrating whenever the notebook closes or disconnected, the cells are not updated anymore.
Could you please add this feature soon!

[AvivSham]

I face the same issue it is a major problem when you run a script which takes several days to run using a server. Sometimes I didn't have a choice and run the script from the beginning all over again. Can you @willingc please address that? The first issue reported in 2015 we are in 2019 and I didn't see any answer. Take for example google colab when you refresh the page still the running cell keeps updating.

[willingc]

@AvivSham Please open an issue on the jupyter/notebook repo or the jupyterlab/jupyterlab repo with details of your particular configuration. Thanks.

[sanxchep]

schrodinger's kernel lmao

Whether the process is running as intended or not

[apayne97]

here to keep the issue alive :). same problem. guess they will never fix it lol.

[alexgarel]

If someone propose a PR, I think they will accept it, indeed !

[halqadasi]

We are in 2022, the problem is still unsolved :(

[sanxchep]

happened again when i was running a model....
why is this a thing in 2022 :(

[ionutscorta]

Just use the following jupyter nbconvert <notebook> --to ipynb --execute --debug in a terminal with tmux active. This will run in the background and plot everything you have in the notebook. The downside is that the notebook is closed when it's done but the good side is you get the plots and metrics from your notebook and if you write it correctly you get the checkpoints from the model.

Annoying issue but who knows when it will be fixed.

[FrankFundel]

You can also use weights and biases for logging.

[nershman]

There's a simple workaround using ipython %%capture magic.

%%capture cap
#Whatever you want to run goes here

#save it to a file
with open('cell_output.txt', 'w') as f:
    f.write(cap.stdout)

When you re-open your notebook you can either load the text file or just print the variable you saved output to:

Not ideal since it doesn't display the contents as it runs, but better than nothing I suppose.

[talha1503]

This problem is still unsolved :/ It's difficult to use jupyter with remote environments, having to keep it open all the time.

[AlexanderPoone]

This bug happens all the time.

[tommedema]

Is this still being considered?

[YogiOnBioinformatics]

PLEASE CREATE A SOLUTION FOR THIS!!!!!!!!!!!

[MichalRIcar]

Hello, same beg from our side - remapping running jupyter server from our server and opening/closing jupyter notebook remotely to check print (log) on our local cause 0 continuations (no progress/prints/logs etc.)..thus for the real-world application NTB is unusable, yet, it is a great "app", it would be great to add this flexible/dynamical element if possible. Or at least a workaround to it.

[rgbkrk]

I've been working on a platform to solve exactly this, called Noteable. You can try it yourself at https://app.noteable.io/

The notebook outputs are updated even if your tabs are closed. You can also open multiple tabs and work on the same document with yourself or others. No worries about clobbering your work and you can let long-running jobs run. Reach out if you'd like to get wider support at your lab or organization.

Hopefully someday we can bring back this tech into Jupyter open source.

[c0d3rman]

Here we are, 7+ years later, still literally zero progress on this. Every single person who's ever used a Jupyter notebook for ML has run into this problem and many lost major amounts of progress because of it. We are not going to stop asking for this until you fix it. It's not a nice-to-have, it's the biggest problem in Jupyter notebooks today, and it needs to be fixed. Does the dev team just not care?

[sanxchep]

Is this application even maintained nowadays?

[axel588]

same issue 7 years later, what a shame

[chunqiaohu]

您好，您的邮件我已经收到，会尽快给您回复，谢谢。

[apayne97]

same issue 7 years later, what a shame

I kind of love the occasional messages I get from this thread though

[thoangtrvn]

same issue for me, when running the notebook with port-forwarding.

[sambbhavgarg]

same issue 7 years later, what a shame

I kind of love the occasionally messages I get from this thread though

frfr, if the devs have not taken it up for 7y, theyre probs not gonna do it now. The best solution so far is to simply update a text/csv with progress every epoch/iteration, and move on...

[sambbhavgarg]

I've been working on a platform to solve exactly this, called Noteable. You can try it yourself at https://app.noteable.io/

The notebook outputs are updated even if your tabs are closed. You can also open multiple tabs and work on the same document with yourself or others. No worries about clobbering your work and you can let long-running jobs run. Reach out if you'd like to get wider support at your lab or organization.

Hopefully someday we can bring back this tech into Jupyter open source.

dude literally created a tool out of a github issue, hats off @rgbkrk ill totally check this out

[xnohat]

For anyone looking for a quick workaround:

• Open the Chrome dev tools
• Go to network tab
• Click WS (Websockets)
• Click on any of the active websockets
• Go to messages tab
• Click on any of the messages after the first
• Expand content
  You should see your output there

another beautified output workaround . Use workaround above with this script in chrome dev tool console

//copy current websocket and change session_id by padding ...-2
const socket = new WebSocket('ws://jupyterdomain.com/api/kernels/c0d04b6d-4a6b-4e21-9e5a-52d4cd7c93f7/channels?session_id=a6ac43c6-4211-4699-b1da-43af501df1b1-2');

// Listen for messages
socket.addEventListener('message', (event) => {
    //console.log('Message from server ', event.data)
    console.log(JSON.parse(event.data).content.text)
});

[thoangtrvn]

I also found a workaround solution. Just don't refresh the browser tab. If you're using Jupyter lab, there are multiple panes,
just close the pane for that notebook, and then on the left side, click to reopen the notebook again.

[362227]

7 years lol

[Wu-yikun]

Same Issue

[jfs42]

super lame this wasn't addressed, guess it's time to stop using Jupyter lab...

[ivanov]

Hello, fellow humans,

There are 68 participants here, but I am going to close this issue, since it has long ceased to be productive. There are some good gems sprinkled about, like [the mention of the %%capture magic](#83 (comment) -1122946511), but the tone of most messages on this issue come from a misplaced sense of entitlement, something to the effect of:

What a bunch of jokers these Jupyter developers, they made and continue to work on this tool we benefit from, make it available for free with a permissive license, but when we tell them about a known limitation of how it works, without contributing anything other than periodic "we still want this" message, they don't do anything about it.

~ Trust me - if it was easy within the current framework within Jupyter, we would have done it by now. And if you don't trust me, no one has stopped you from trying to do it yourself. But piling on with a "bump!" or "PLEASE FIX THIS!" doesn't do anything
(except maybe chip away at my faith in the human spirit). Think about this: is it more sad that this situation is perceived to be the same as it stood eight years ago, or that so many people keep coming here to demand their outputs of long running notebooks without pitching in themselves?

I also worked on the Noteable platform. And no, @sambbhavgarg, though he does have superhuman hacking powers, @rgbkrk didn't create it by himself :) , there were about a dozen people working on it, and no, it wasn't created just to solve this one issue. Importantly, the way it did get solved there involved a significant engineering effort to move the state of the notebook from the browser to the server, which is one of the approaches that's also been happening in parallel out in the open source community.

Those of you wanting to follow along and interested in seeing this addressed in Jupyter are encouraged to follow these issues, or use them as starting points to getting involved.

jupyter/notebook#641
jupyterlab/jupyterlab#2833
jupyter-server/jupyter_server#1274
https://github.com/jupyter-server/message_replay

I'll be closing this issue, but if you want to continue the discussion, you can do so over on the Jupyter Discourse.
Happy hacking!