Skip to content
Utility files to help set up colab for experimentation + development
Branch: master
Clone or download
Type Name Latest commit message Commit time
Failed to load latest commit information.


Utility files to help set up colab for experimentation + development

The idea is to let you :

! git clone
from colab_helper import utils as chu

at the top of a notebook, and have a bunch of useful stuff ready-to-go (you can choose the name under which to import it, so as to avoid collisions with your existing code).

Google Drive helper

This mounts your Google Drive at (per convention) ~/gdrive but also optionally adds a link, so that you can use a path that doesn't need the awkard space character introduced by 'My Drive' :

chu.gdrive_mount(point='gdrive', link='my_drive')
! ls -l my_drive/*


This just cleanly downloads data (unwrapping by default), without downloading (or unwrapping) when the required files are already present.

Single file (no unwrap required):''

More complex .tar.gz example (the dest_path parameter allows it to check on whether the unwrapped files have appeared in a particular directory) :''

Kaggle Credentials helper

Generate the kaggle.json file and upload it to Colab, or just use your username and key in-line :

! pip install kaggle

Then you can access the Kaggle CLI (see also the Kaggle API docs):

# Description page :
! kaggle datasets download ronitf/heart-disease-uci

SSH Reverse Proxy

This is for expert use only. If you don't know what this is doing, or how to get it to run, then this isn't something you should be messing with.

Note also that this is far more security conscientious than other scripts you might find on the web : It doesn't allow logins via passwords, for instance, nor execute arbitrary scripts downloaded from a url.

Example use (it will print out the required local ssh command) :

ihku00gbBwSOu2M38GMdGV9qU9XrEkLSjD/1WtzYJZL7buzpitlGlTvhnqQT+t andrewsm@square.herald

The pub_key field cleans out any line-breaks pasted in from ~/.ssh/ for your convenience.
And, as an aside, there's no problem leaving your public key(s) in the colab file itself, since that's not the private key bit (obviously).

Using the rsync command given in the output, one can then do a auto-sync-to-colab for locally edited files (use the %autoreload 2 magic to transparently have the updated code reloaded as you run the notebook cells) :

while rsync-command-from-colab_helper; do inotifywait -qqre close_write,move,create,delete code/; done

File thinning

To reduce the number of saved checkpoints to 3 recent ones, plus 7 others with 'round numbers', simply :

! git clone
from colab_helper import files as chf

chf.thin_numbered_files('./checkpoints/2019-07-26_01-clipnorm', delete=True)
# This will return a dict(keep=?[], ?delete=[], ?comment=''), 
# and if 'delete=True' is passed in, the 'delete' array will have been removed (careful!)

There are obviously more options, but the simple library is intended to 'do the right thing' (and returns its suggestions if delete=False, which is the default value). For instance, it should be able to figure out which filenames form a series (with the longest string prefix), and then extract the epoch/step numbers. Of course, it's assumed that the files are saved using some kind of convention, with the epoch/step as the first group of digits, like :

# ...
# ...
You can’t perform that action at this time.