Testing elasticluster on GCE

These are notes on running the tutorial document sent to Hyun by the GCE team.

Goal

Start and excercise a cluster of machines with SGE using GCE.

Workflow:

Create a virtualenv:

$ virtualenv elasticluster  
$ source elasticluster/bin/activate  

Install an older setuptools (pip):

$ pip install setuptools==9.1  

Install an older ansible (pip):

$ pip install ansible==1.7.2  

Install elasticluster (from github):

$ cd elasticluster
$ git clone https://github.com/mbookman/elasticluster.git src
$ cd src
$ python setup.py install
$ pip uninstall --yes google-api-python-client
$ pip install google-api-python-client

Modify the minimal template. Set gce_project_id, gce_client_id, gce_client_secret, and image_id values. The gce_client_id and gce_client_secret are set through the credentials tab in the Google Developer console. I have not uploaded the full config as it contains sensitive information.

Copy the modified config to ~/.elasticluster/config. In our case, we are using a three node cluster: one gateway, and two workers. The VMs are based on debian wheezy.

Start a cluster:

elasticluster start -v gridengine

SSH into the gateway:

elasticluster ssh -v gridengine

Run a short job that exercises the nodes.

Stop the cluster:

elasticluster start -v gridengine

Success!

I am able to spin up a small cluster with the Google Compute Engine.

Notes:

• Matt Bookman also provided instructions for installing a google-fork of elasticluster, and the pip version. Not sure what the recommended practice is.