Files Disappear When Still Running

[chuktuk]

Bug report for Colab: http://colab.research.google.com/.

For questions about colab usage, please use stackoverflow.

• Describe the current behavior:
  I'm training a Flair NLP model. This is an RNN that requires a lot of time and memory, so I'm using paid Pro version. I tried the free version, and it would time out before the model could finish training.

When training is complete, a file containing the best model is output. I saw this file created originally during model training, however, after a few hours, in the file area it only says 'Connecting to a runtime to enable file browsing.' It will be very annoying and a waste of money if I can't retrieve the best model file after all this time.

• Describe the expected behavior:
  The files area should not time out when the model is still running! If it does, then it should be intuitive to recover the files.

• The web browser you are using (Chrome, Firefox, Safari, etc.):
  Chrome

• Link to self-contained notebook that reproduces this issue
  (click the Share button, then Get Shareable Link):
  https://colab.research.google.com/drive/1nzQ9jBGlokVswHTWS-6prQEjdPmkQ0Sk

[chuktuk]

When this happens, the button that usually has the RAM and CPU usage bars just says 'Busy'.

[tnovikoff]

It sounds like you may be running into the resource limits (max vm lifetimes or idle timeouts) of Colab Pro.

Information about resource limits in Colab Pro can be found in the Colab Pro FAQ at http://colab.research.google.com/signup

Information about resource limits in Colab in general can be found in the main Colab FAQ at
https://research.google.com/colaboratory/faq.html

Information for getting the most out of Colab Pro can be found at
https://colab.research.google.com/notebooks/pro.ipynb

One specific approach that might work in your situation is saving your model file out to Google Drive while the connection is active.

More generally, this might be a good question for StackOverflow, where other users may be able to provide other ideas for what to do in this situation.

Best of luck!

[chuktuk]

I couldn't find anything on stack overflow regarding this type of issue. I found this link on Medium https://medium.com/@ml_kid/how-to-save-our-model-to-google-drive-and-reuse-it-2c1028058cb2 about saving in Google Drive, however, it doesn't explain HOW to save after every epoch. I'm using Flair on top of PyTorch, and it is creating the file automatically. I also don't see how I could start in the middle if it crashed after an epoch. Unless I find a solution to this, it looks like the paid version doesn't have enough resources for my needs.

[tnovikoff]

You could try asking your question on StackOverflow. :-) Best of luck either way.

[dexter2406]

When this happens, the button that usually has the RAM and CPU usage bars just says 'Busy'.

This happens to me too, do you have any solution now?

[chuktuk]

I ended up having to start the model very early and clicking in various cells from time to time until the model finished training. I then immediately saved the model files to generate my predictions. I tried a javascript function to click cells, but that didn't seem to work. The only solution I found was to keep the window active. Maybe AWS is a better option, since you can pay for time instead of doing a subscription like the paid Colab version I used. Best of luck.

…

On Thu, May 6, 2021 at 7:05 PM dexter2406 ***@***.***> wrote: When this happens, the button that usually has the RAM and CPU usage bars just says 'Busy'. This happens to me too, do you have any solution now? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1045 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK47M6JCT5TI2IMPOWFMFQDTMMODHANCNFSM4LBVMKQQ> .

-- Charles R. Tucker Sr. Data Analyst Data Management Santee Cooper

[stmer1]

I had this problem too, though for me, I found that if I hit Runtime>Interrupt execution, everything magically reappeared. I did not try restarting, since I decided it had probably trained enough, and I wanted to make sure to grab what it had done so far. Not the ideal situation, but at least I did not lose everything.

Good luck.