Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keras progress bars cause jupyterlab to freeze firefox and chrome #4202

Closed
ogrisel opened this issue Mar 18, 2018 · 21 comments · Fixed by #6304
Closed

Keras progress bars cause jupyterlab to freeze firefox and chrome #4202

ogrisel opened this issue Mar 18, 2018 · 21 comments · Fixed by #6304

Comments

@ogrisel
Copy link

@ogrisel ogrisel commented Mar 18, 2018

I noticed the builtin keras progress bar often causes the jupyterlab UI to freeze after a while (both under firefox and chrome, under Linux at least). I do not observe this issue under the traditional jupyter notebook app.

Here is a minimal reproduction script that does not depend on Keras. I am not sure Keras is implemented that way but might script seem to be able able to reproduce the same behavior as Keras is causing.

import time


def make_progress(n_steps=int(1e4), line_length=70, delay=0,
                  flush_every=1e-2, prompt=""):
    last_flush = time.time()
    for i in range(n_steps):
        progressed = ((i + 1) * line_length) // n_steps
        prefix = '#' * progressed
        suffix = '-' * (line_length - progressed)
        now = time.time()
        flush = (now - last_flush) > flush_every
        if flush:
            last_flush = now
        print(f'\r{prompt}{i + 1:05d} [{prefix}{suffix}]', end="", flush=flush)
        if delay:
            time.sleep(delay)

            
for i in range(30):
    make_progress(prompt=f"{i + 1:02d} ")
    print()

Steps to reproduce:

  • copy the above snippet in a jupyterlab cell in a fresh new Python 3.6 notebook
  • execute the cell

Observed:

  • the progress bars display one after the other;
  • the first progress bars are very smooth;
  • after 5 or 10 bars, progression is much less smooth;
  • after 20 or so firefox window is completely frozen: it's clicking on the interrupt button takes a while to be taken into account.

Expected:

  • smooth progress bars as observed when running the same notebook in the old jupyter notebook instead of jupyterlab.

I suspect that the \r not handled in the same way as in jupyter notebook and some datastructure in the browser is growing beyond acceptable.

Note that this bug is not the same as #1104 that was already fixed in previous versions of jupyter lab.

This issue was observed using jupyter lab 0.31.10. I have just upgraded to 0.31.12 and it seems to behave a bit better: I now get the following warning after a while:

IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_data_rate_limit`.

Current values:
NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
NotebookApp.rate_limit_window=3.0 (secs)

but firefox still gets much slower than under the original jupyter notebook app after a while.

@ogrisel
Copy link
Author

@ogrisel ogrisel commented Mar 18, 2018

Actually, I also observed the IOPub data rate exceeded. warning on jupyter notebook.

@ogrisel
Copy link
Author

@ogrisel ogrisel commented Mar 19, 2018

Maybe it's not that a big issue anymore in the recent versions. There might still be opportunities for further optimization of transfer of text streams with frequent calls to flush that also have many \r occurrences. Maybe using a server-side buffer in case of too frequent calls to flush for instance.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Mar 19, 2018

Maybe it's not that a big issue anymore in the recent versions.

Thanks. So are you saying JupyterLab behaves like the classic notebook right now?

The server-side buffer idea would be an issue for the ipython kernel or the jupyter notebook server.

@jasongrout jasongrout added this to the Reference milestone Mar 19, 2018
@ogrisel
Copy link
Author

@ogrisel ogrisel commented Mar 19, 2018

Yes I believe so but both seem to slow down when running the above snippet.

@sccolbert
Copy link
Contributor

@sccolbert sccolbert commented Mar 19, 2018

but firefox still gets much slower than under the original jupyter notebook app after a while.

This is not too surprising. The updates are likely causing a layout reflow of the notebook, which we know is slow on FF and will be fixed for 1.0

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Mar 19, 2018

For what it's worth, I tried the example above in current jlab master and didn't see any slowdown across the 30 progress bars. FF 59 on macOS 10.13.3.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Mar 19, 2018

cf also #1639 for firefox rendering/layout slowdown issues.

@MrMathias
Copy link

@MrMathias MrMathias commented Mar 27, 2018

I am also experiencing freezes using chrome/windows. Does this have to do with the update rate or the absolute number of updates of the bar? - I have also experienced that the bar itself just stops progressing, might be related.

@uzi0espil
Copy link

@uzi0espil uzi0espil commented Jul 17, 2018

I am facing same issue.. Running a keras script on Chrome browser v. 67.0.3396.99, After some epochs, the browser hangs and asks for killing the page. I tried also with incognite and still same problem

Jupyterlab version: 0.32.1
OS: Ubuntu 16.04

@wernight
Copy link

@wernight wernight commented Aug 12, 2018

Could have a bit less output, but not nothing like verbose=0. This may "solve" it for most cases.

@sebastianffx
Copy link

@sebastianffx sebastianffx commented Nov 14, 2018

Same issue here. Ubuntu 16.04, Chromium 70.0.3, Jupyterlab 0.35.4, Keras 2.2, python 3.6
My workaround so far is just to check that training starts OK for some epochs and then stop it, download the notebook as a python script, and execute it directly on the terminal.

@atamborrino
Copy link

@atamborrino atamborrino commented Nov 27, 2018

Same issue here. I can not train a Keras model for more than 5 epochs... IMHO this is a blocking issue for using JupyterLab with Keras.

@ghostandthemachine
Copy link

@ghostandthemachine ghostandthemachine commented Dec 6, 2018

I've had this same problem. My workaround was to set verbose=0 in the fit call then also add a LambdaCallback which updates an IPythonWidget IntProgress bar so I get the same info. I think this is actually a nicer solution then the long print statements. I pass the display elements to my callback creation fn:

def _progress_cb(progress, html):
    
    def on_epoch_end(batch, logs):
        progress.value += 1
        html.value = f"<div><div>Epoch: {batch}</div><div>{logs}</div></div>"
    
    return callbacks.LambdaCallback(
        on_epoch_end=on_epoch_end
    )

Then use it with something like:

html = widgets.HTML(value="")
progress = widgets.IntProgress(min=0, max=num_total_epochs, value=0)

display(progress)
display(html)

model.fit(x=x, 
          y=y,
          verbose=0,
          callbacks=[_progress_cb(progress, html)])

You obviously loose history in your cell output div but I find TensorBoard more useful for that anyway.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Dec 6, 2018

I've had this same problem. My workaround was to set verbose=0 in the fit call then also add a LambdaCallback which updates an IPythonWidget IntProgress bar so I get the same info. I think this is actually a nicer solution then the long print statements.

Nice! Thanks for sharing!

@dbuades
Copy link

@dbuades dbuades commented Feb 5, 2019

Same issue here, though it works perfectly with the classical jupyter notebook.

@divamgupta
Copy link

@divamgupta divamgupta commented Apr 26, 2019

Hi, any updates regarding this issue?
The initial versions of jupyterlab did not have this issue

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Apr 26, 2019

Is this still an issue with the 1.0a3 released last week? There has been some recent work on handling \r, which is what these progress bars are using, IIRC.

@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Apr 26, 2019

Yes, this is still an issue in 1.0a3. The code in the OP works great for me in classic notebook, ff 66, macOS. Stops dead in the first progress bar in JupyterLab 1.0a3, ff 66 or chrome 74.

I'm going to tag it for 1.0 and try to spend at least some time looking at it, to see if there is an obvious optimization we're missing somewhere.

@jasongrout jasongrout removed this from the Reference milestone Apr 26, 2019
@jasongrout jasongrout added this to the 1.0 milestone Apr 26, 2019
@jasongrout jasongrout self-assigned this Apr 26, 2019
@jasongrout
Copy link
Contributor

@jasongrout jasongrout commented Apr 26, 2019

@ian-r-rose has some thoughts on this, like that the classic notebook does some update throttling.

@ian-r-rose
Copy link
Member

@ian-r-rose ian-r-rose commented Apr 26, 2019

Specifically here

@blink1073
Copy link
Member

@blink1073 blink1073 commented May 5, 2019

Thanks to @ian-r-rose for the fix! I just released the fix in 0.35.6. I verified by running @ogrisel's original example.

@lock lock bot locked as resolved and limited conversation to collaborators Aug 7, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.