colab using too much local memory when reloading web page while model is training

[HAadams]

Bug report for Colab: http://colab.research.google.com/.

For questions about colab usage, please use stackoverflow.

• Describe the current behavior:

  • If I clear the logs using the "X" button under a cell while it is running (training a DETR model) and then I refresh the colab notebook page, the website starts freezing and consumes over 8gb of local ram. Note that the notebook is running on Google's hosted environment, not local environment.
  • The notebook unfreezes and becomes usable again if I either
    1. Terminate the notebook from another session
    2. Or wait for the notebook to finish training or time out

• Describe the expected behavior:

  • Clearing the logs of a running cell and then refreshing the colab notebook page should not cause usage of over 8gb of local ram.

• The web browser you are using (Chrome, Firefox, Safari, etc.):

  • Chrome, but happened on Safari as well and Windows edge browser
    

• Link (not screenshot!) to a minimal, public, self-contained notebook that
  reproduces this issue (click the Share button, then Get Shareable Link):
  The dataset used by the notebook is stored in my google drive so you will not be able to run the notebook.
  https://colab.research.google.com/drive/1DD6kxCczXkAKl1vwD_Z10wo9JMLXSyZ8?usp=sharing

[HAadams]

Could this be due to too many log messages?

[blois]

Can you also try an incognito window?

I'm curious if you know what sort of outputs are occurring- just regular console logging or anything like tqmd_notebook? This seems similar to #1708.

[HAadams]

Hello, same issue in incognito as well. Also, looking at the DETR code, they use regular print() statements so the outputs are just regular console logging. These models spit out a lot of logs. I've noticed that at some point, the amount of logs accumulated reaches a point where the Google Colab notebook page freezes on its own OR slows down tremendously. When I catch it slowing down, I clear the output and then refresh the page as an attempt to recover it but that causes it to not load until the model is done training or the notebook timeout.

I'll try to repro with
while True:
print("insert random stuff here"*200)

[HAadams]

just an update, I tried to run the following code on a new notebook:

import time
while True:
  time.sleep(1)
  print("saf asdf asdf hello"*100)

After 15-20 minutes, clearing the output then refreshing the page causes the site to freeze momentarily. But since the output is not too large, the page loads after sometime. I'm thinking if I leave the while loop running overnight, it'll cause the same issues I saw with the notebook freezing until it either timeout or finishes running.

[blois]

In the cell which is running for a long period of time, I'm curious if you could add at the top:

from IPython.display import HTML
display(HTML(''))

So the example from #1746 (comment) would be:

from IPython.display import HTML
display(HTML(''))

import time
while True:
  time.sleep(.01)
  print("saf asdf asdf hello"*100)