Switch branches/tags
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
baseliner
geni
README.md
run.sh
setup.sh
teardown.sh

README.md

CloudLab ready pipeline

The CloudLab  ready badge is included in a repository hosting a Popper pipeline that is ready to be executed on CloudLab. If you see it in a repository, it means that the authors have made it easier for others to run on CloudLab by following the Popper convention.

In this repository, we present an example of a pipeline that can be used as the starting point to implement a CloudLab  ready pipeline. To add it to your repository using popper, you can do:

popper add popperized/popper-readthedocs-examples/cloudlab-benchmarking

If you add this pipeline to your repo (or if you follow the Popper convention to implement experiments on CloudLab), we invite you to add the badge to your README file.


CloudLab is a flexible, scientific infrastructure for research on the future of cloud computing. It gives students and researchers access to cloud resources so they can setup clusters of machines to execute experiments in many domains such as networking, computer systems, cloud systems, etc. You can check the CloudLab Manual here.

This pipeline is an example of a CloudLab  ready pipeline, i.e. a pipeline that can be executed on CloudLab with minimal effort. The pipeline consists of 3 stages:

  • setup. This stage automates the allocation of nodes on CloudLab using the geni-lib library (specifically this docker image). The stage requests as many nodes as the ones specified in the request file. At the end of this stage, a list of available nodes is written to geni/machines.
  • run. Invokes Baseliner in order to execute the containerized benchmarks specified in baseliner/config.yml on each of the machines from the geni/machines file. At the end of the execution, the folder results/baseliner_output holds the output of the containers, sorted by benchmark and machine.
  • teardown.sh. Releases all the nodes that were allocated on the setup stage.

Manual execution

To manually execute this pipeline:

  1. Get the pipeline by executing the following command:

    popper add popperized/popper-readthedocs-examples/cloudlab-benchmarking
  2. Obtain credentials (see here). This will result in having a cloudlab.pem file on your machine.

  3. Define the following environment variables.

    • CLOUDLAB_USER. Your user at CloudLab.
    • CLOUDLAB_PASSWORD. Your password for CloudLab.
    • CLOUDLAB_PROJECT. The name of the project your account belongs to on CloudLab.
    • CLOUDLAB_PUBKEY_PATH. The path to your SSH key registered with CloudLab.
    • CLOUDLAB_CERT_PATH. The path to the cloudlab.pem file downloaded in step 2.
    • SSHKEY. The path to your private SSH key registered with CloudLab.
  4. Execute the command:

    popper run cloudlab-benchmarking

Automated execution (CI)

To execute this pipeline on a CI system:

  1. Define the following hidden (secrets) environment variables on your CI system:

    • CLOUDLAB_USER. Your username at CloudLab.
    • CLOUDLAB_PASSWORD. Your password for CloudLab.
    • CLOUDLAB_PROJECT. The name of the project your account belongs to on CloudLab.
  2. Encrypt the following files using your CI system utilities and move them to the geni/ folder:

    • cloudlab.pem. File downloaded in the previous stages.
    • id_rsa.pub. Your public ssh key registered with CloudLab.
    • id_rsa. Your private ssh key registered with CloudLab.

TravisCI

We now explain how to configure TravisCI so it automatically validates the pipeline. Before starting, make sure you have done the following:

  • Installed the Travis CI Command Line Tool by running:

    gem install travis
  • Logged in to using the TravisCI CLI tool:

    travis login
  • Define secrets from step 1 from above on Travis.

Once the above is done, on your machine do the following. Compress the files from Step 2 of the previous to a file called cloudlab_credentials.zip or similar. Then, execute the command:

travis encrypt-file cloudlab_credentials.zip --add

This will generate a cloudlab_credentials.zip.enc file and will add to your .travis.yml file something like:

openssl aes-256-cbc -d -K $encrypted_0a6446eb3ae3_key -iv $encrypted_0a6446eb3ae3_iv -in cloudlab_credentials.zip.enc -out cloudlab_credentials.zip

Please make sure to correct the path of files, for example:

openssl aes-256-cbc -d -K $encrypted_0a6446eb3ae3_key -iv $encrypted_0a6446eb3ae3_iv -in /route/to/cloudlab_credentials.zip.enc -out pipelines/cloudlab-benchmarking/geni/cloudlab_credentials.zip

It is important to verify that the file cloudlab_credentials.zip is in pipelines/cloudlab-benchmarking/geni/ as in the previous example. This is where the pipeline reads the decrypted files from.

To unzip it as part of the pipeline, add the following to your travis file.

  - unzip pipelines/cloudlab-benchmarking/geni/cloudlab_credentials.zip -d pipelines/cloudlab-benchmarking/geni/

To see the final result, you can check the .travis.yml file for this repo. Lastly, commit all changes to your git repository and push to github so that Travis executes your CloudLab ready pipeline.

NOTE: Make sure to add cloudlab_credentials.zip.enc to the git repository and NOT the clouadlab_credentials.zip file to your git repository, otherwise all your credentials will be available on github to anyone with access to your repo.