Google Compute Engine ready pipeline example
Google Compute Engine (GCE) is a service offered by Google to facilitate the deployment of cloud infrastructure. It provides a well tested and easy way to allocate resources on Google's global infrastructure, and gives anyone the ability to painlessly create, run, and destroy virtual machines at will.
This directory contains a pipeline that makes use of GCE to run a simple experiment. We use Terraform to programatically create and destroy the compute instances needed for the experiment.
The pipeline consists of three stages:
setup.shinitializes Terraform and applies the changes needed to create the infrastructure we need. To avoid requiring terraform to be installed, we instead use a the official Terraform docker image provided by Hashicorp. Once the machines are ready, a list of them is written to
run.shruns Baseliner on the provisioned machines and saves results to a
teardown.shonce again uses Terraform, this time to free the resources we created in out setup stage.
To run this pipeline locally:
- Add the pipeline to your current repository with this command:
popper add github/popper-readthedocs-examples/gce-benchmarking
- Place your Google Cloud service account credentials json file inside the
credentialsdirectory of the pipeline as
account.json. If you don't have a service account, you can find more information on obtaining one here.
- If required, you may set the
GCE_MACHINE_TYPEenvironment variable to change the size of the instance running the experiment. The default value is
- Execute the pipeline:
popper run gce-benchmarking.
Automatic execution in continious integration evnironments
Setting this pipeline to run on top of a continious integration service is similar to the standard procedure for most popper pipelines, described here. The only added complication is the need to have the
credentials/account.json file present during runtime. You should not ever commit bare account credentials to a public repository of any kind. Instead you should take advantage of features offered by various popular CI services to commit an encrypted version of your credentials file in the repository instead. A few examples:
- The Travis CI documentation has a section on encrypting files
- Circle CI has a recommended way of keeping sensitive files encrypted in your repository