aws-magic

Based on the fastai course.

The point of this repository is to abstract away aws CLI complications. By tying some state on the AWS cloud to your laptop, cloud machines can be a one-liner away. All action centers around directories in the following managed location in your home directory:

~/aws-instances

If you launch a machine with name name, then this directory will be created on your computer:

~/aws-instances/name

That directory will have a couple of scripts that give the simplified interface to AWS you wish you had, it's like magic!

~/aws-instances/name/start  will start up the cloud machine
~/aws-instances/name/ip     will print the public IP of the running instance
~/aws-instances/name/ssh    will ssh into the running instance
~/aws-instances/name/stop   will stop the currently-running instance
~/aws-instances/name/state  will describe the current state of the instance
~/aws-instances/name/delete will stop the instance and delete all metadata about the instance from both your computer and AWS

You're only billed on EC2 when the machine is in running state, but even if you stop the machine you'll still be charged for the cost of storing the stopped machine image on EBS.

Launching a Machine

Quick-start:

./aws-oregon.sh

Requires that the default aws cli user has been set up (i.e., ~/.aws/config should have your API keys). This is a thin wrapper with defaults around:

setup-instance.sh ami instancetype name open-extra

This will create and start a cloud machine on AWS. Security info:

• Ports 6006 (Tensorboard), 8888-8898 (Jupyter), and 22 (ssh) are open if open-extra is true. If it's false, only ssh is open.
• ssh is only accessible by a public key stored in your ~/.ssh
• No Jupyter process is started on the server by default. If you do run Jupyter, I suggest following these guidelines for security.

Cleanup. If you send a ctrl-C during the script, or there is an issue during resource allocation (invalid instance type or instance limit hit), then as long as you don't freak out and send another quit/terminate signal the script will attempt to cleanly de-allocate all reserved AWS resources. After completion the local folder ~/aws-instances/name for server with name name is deleted. If stuff really goes wrong then you might need to manually log onto the AWS web interface to clean things up, but that shouldn't happen through normal use.

Installing Deep Learning Software

If you have a live instance running (generated by a script such as the above), then you can install my deep learning setup on it (provided you want TensorFlow, Keras, my .bashrc, etc.). See server/server-install-deeplearn.sh for details, and to start the install on your laptop use:

./interactive-server-install-deeplearn.sh

Note that this is not well-polished and has lots of details specific to me right now, use at your own risk. This is intended to install on top of Ubuntu 16.04 and add tools for interactive development. This probably has lots of features not necessary for just running existing code.