example dev setup environment using librarian, chef and vagrant
Ruby
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
.gitignore
Cheffile
README.md
Vagrantfile

README.md

Table of Contents generated with DocToc

Tested With

This setup has been tested with:

  • OSX 10.7.5
  • vagrant v1.2.7
  • virtualbox 4.2.16
  • chef 11.6.0
  • ruby 2.0

Overview

This is a sample repo for spinning up a dev vm. We use librarian, Vagrant and chef solo and we will end up with a virtual machine with a number of python packages installed, running an instance of iPython Notebook that can be accessed from the host browser via port forwarding.

We also include instructions on how to create your own custom chef cookbooks that can be added in to the setup.

Quickstart

If you have already have Vagrant, chef solo and librarian installed, and someone has configured their environment as described then all you need to do to get started is

$ git clone git://github.com/elifesciences/elife-template-env.git
$ cd elife-template-env
$ librarian-chef install
$ vagrant up
$ vagrant ssh
$ ipython notebook --ip=192.168.33.10

Then in your host browser open http://192.168.33.10:8888.

Dependencies

You need to have Vagrant, chef solo and librarian installed. The following instructions are for the bash shell.

Installing chef

From the opscode wiki

sudo true && curl -L https://www.opscode.com/chef/install.sh | sudo bash

Installing librarian

Librarian is a ruby gem that manages chef recepies.

$ gem install librarian

Installing and configuring Vagrant

Vagrant is a wrapper to a virtual box VM. There is plenty of documentation. You can get Vagrant here.

Vagrant needs a base box defined. Once defined it should download the machine image that will get reused. A nice list of images can be found at http://www.vagrantbox.es/. We are currently recommending using http://files.vagrantup.com/precise64.box.

After installing vagrant configure it as follows

$ vagrant box add precise64 http://files.vagrantup.com/precise64.box

Preparing a new bare repo in git that will be configured via librarian/chef

From within your bare repo initialise Vagrant.

$ vagrant init

This will generate a local Vagrantfile.

Now set up the git repo to ignore files that will be created during the running of librarian and chef.

$ echo cookbooks >> .gitignore
$ echo tmp >> .gitignore
$ git add .gitignore
$ git commit -m "ignoring some files"

Configuration

Configuring chef using librarian

Create a Cheffile using librarian.

$ librarian-chef init

You now co-customise the Vagrantfile and Cheffile to refer to the same cookbooks that you require for your environment. The Cheffile can actually contain more dependencies than you need in your environment if you wish (for example if you have a generic set of tools and you don't want to have custom Cheffiles for each project).

The Cheffile for this project currently looks like:

#!/usr/bin/env ruby
#^syntax detection

site 'http://community.opscode.com/api/v1'

cookbook 'apt'
cookbook 'git'
cookbook 'python', '>= 1.2.2'

cookbook 'rvm',
   :git => 'https://github.com/fnichol/chef-rvm'

cookbook 'build-essential',
    :git => 'http://github.com/opscode-cookbooks/build-essential'

cookbook 'elife-chef-dev-template',
        :git=> 'http://github.com/elifesciences/elife-chef-dev-template.git'

While the appropriate part of my Vagrantfile looks like:

Vagrant::Config.run do |config|
  config.vm.provision :chef_solo do |chef|
  # This path will be expanded relative to the project directory
  chef.cookbooks_path = "cookbooks"
  chef.add_recipe "apt"
  chef.add_recipe "build-essential"
  chef.add_recipe "rvm::vagrant"
  chef.add_recipe "rvm::system"
  chef.add_recipe "git"
  chef.add_recipe "python"
  chef.add_recipe "elife-chef-dev-template"
end

The important thing here is that for any package that you want to include in your VM, Vagrant needs to have that package specified, and the package also needs to be specified in the Chefile.

Some final configurations for Vagrant

If you are running a web app, you want to be able to access the server in the VM from the host machine. This can be done via port forwarding. In addition set a static domain in the Vagrant file

config.vm.network :hostonly, "192.168.33.10"

Once you have configured these to your tastes you can commit them.

$ git add Vagrantfile
$ git add Cheffile
$ git commit -m "env setup"

Obtaining the cookbooks using librarian.

The Cheffile describes which cookbooks we want to use. librarian runs against the Cheffile, gets the cookbooks, and sets them up locally with all path dependencies. Vagrant can then use the local cookbooks for configuring the VM.

$ librarian-chef install

Invoking the VM

Bringing up the dev environment

Fingers crossed!

$ vagrant up

You can access the shared folder between the vagrant file system and the user system through

$ ls ~/vagrant

To stop the VM, in console:

$ vagrant suspend

To remove the VM, in console:

$ vagrant destroy

To reload the VM, in console:

$ vagrant reload

Accessing iPython notebook from the host machine

$ vagrant ssh
$ ipython notebook --ip=192.168.33.10

You can now access your ipython notebook from the host machine by opening a browser at http://192.168.33.10:8888. It's important to pass the ip parameter to iPython, as it needs to be made aware of the endpoint that the VM will be using to communicat out of the box. This obviously needs to be set to be the same as the ip address that was set in the Vagrantfile. Enjoy!

Distributing your project.

You can commit the Vagrantfile and the Cheffile, push the changes to git hub and your dev environment is ready to distribute.

Creating your own cookbook for chef solo.

If you want to add a separate set of dependencies, or cookbook, then in a separate repo you should create a default Chef cookbook using [knife][kf]

$ knife cookbook create cookbook-name

Create a github repo for this cookbook, and then point your Cheffile at the repo, and update your Vagrant file accordingly.

For this demo repo we have created an eLife specific cookbook that installs the following python dependencies:

  • requests
  • lxml
  • beautifulsoup4
  • fom
  • lettuce
  • boto
  • nose
  • ipython-notebook
  • scipy
  • pandas
  • sympy

Provisioning to AWS

Install the vagrant aws plugin

$ vagrant plugin install vagrant-aws

Configure the aws plugin with your info. It seems you need to create an AWS Ec2 instance, nominate the name of the ami, setup your local ssh credentials for accessing this ami, and then give vagrant these details, along with your aws keypairs. A good overview is provided here:

http://www.devopsdiary.com/blog/2013/05/07/automated-deployment-of-aws-ec2-instances-with-vagrant-and-puppet/

Bring up the machine in AWS

$ vagrant up --provider=aws

Gotchas

ports

Vagrant won't shit all over your ports in use. The default Vagrant file in this repo asks vagrant to setup on port 8080, but if you have any other services running on this port Vagrant will not bring up the machine. Either kill your service, or change the ports in the Vagrantfile.

building a big machine

If you build a machine that has a lot of installs, vagrant can hang in the process. It looks like it doesn't work. Treat it like a computer science problem, kick it, turn it off and turn it on again and hope that it works.

TODO

  • provide an example of cloning a repo into the newly created vm
  • pass auth credentials either using a databag or some other method
  • give an example of how to deploy the vm into AWS

Some useful links.

http://docs.vagrantup.com/v1/docs/provisioners/chef_solo.html
http://wiki.opscode.com/display/chef/Resources#Resources-Execute
http://jtimberman.posterous.com/guide-to-writing-chef-cookbooks
http://kallistec.wordpress.com/2010/02/01/the-chef-way-episode-2-chef-speak/
http://wiki.opscode.com/display/chef/Shef
http://blog.base2.io/2012/05/01/vagrants-and-chefs-and-librarians-oh-my/#.USTZpFrfz5Q