Continuous integration and staged deployment.
Ruby Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


Note that Immunity is a work in progress and not yet in production use. It still has Ooyala-specific elements to the code.

Immunity System

Immunity is a web service which orchestrates continuous integration and deployment in stages. It's designed to allow for fast, continuous deployment, while preventing bad code from getting into production and affecting your customers. It's your app's immunity system.

When a new commit is made, Immunity deploys your app to one of its available regions (group of servers), runs integration tests, and optionally monitors application metrics while receiving mirrored production traffic.

Workflow example

Here is a concrete example of a good Immunity workflow for a production web server called "Papi". The example illustrates how you can configure Immunity to power your deployment pipeline.

Say that Papi is configured with four regions, each of them a single machine: Sandbox1, Sandbox2, Prod3, Prod4. The first two are used for development and do not receive production, whereas the second two do.

"Application metrics" are collected using a monitoring system (like Hastur). They are metrics like error rates, latency, SQL query count, memory usage, successful signups, etc. While running, Papi reports its metrics to that metrics system, and Immunity is set up to read those metrics.

  1. A new commit is made to Papi.git, which causes Immunity to deploy this version of Papi to Sandbox1.
  2. Papi's integration tests are run on Sandbox1.
  3. If they pass, Papi is deployed to Sandbox2.
  4. Papi's integration tests are run on Sandbox2.
  5. If they pass, Sandbox2 starts receiving mirrored readonly production traffic from Prod3.
  6. After 5 minutes, the application metrics recorded on Sandbox2 are compared to the metrics recorded on Prod3. If they are within an acceptable range, this version of Papi is deemed good and ready for production traffic. In the Immunity UI, this version of Papi can now be deployed to prod3 (which does receive production traffic) by the click of a button.
  7. When that button is clicked, this new version of Papi is deployed to Prod3.
  8. For each remaining server (e.g. Prod4), the server is deployed to, has its integration tests run, and app metrics monitored, one server at a time.

In this example, Sandbox1 is used to provide the same workflow as a traditional continuous integration server. New versions of Papi are deployed right away, tests are run, and if something fails, developers are notified. Feedback for test breakages is quick and clear.

Sandbox2 gets deployed to less frequently, because the step of monitoring the app's metrics takes longer. This step is more realistic and safe and is used to verify that the new version of Papi will perform well in production. It's used to detect more subtle breakages under real customer usage and load.

Why Immunity

Building a continuous, staged deployment workflow for your app is hard and always application-specific. Immunity makes it easier to script these workflows with these features:

  • A basic workflow engine that's specific to deploying versions of code to groups of servers, including "monitoring application metrics".
  • A UI which succinctly visualizes the current state of your deployment pipeline along with metrics useful for troubleshooting.
  • A focused and easy-to-extend code base.

These features reduce a lot of the complexity needed for scripting a robust continuous deployment and ensure that you don't have to beat your infrastructure into submission to get it to do what you want.

Why not use something like Jenkins for this?

Jenkins is a generic job server which can be used to run jobs like "deploy version X" or "run the tests for version Y". However, since it is a generic job server, it takes a lot of configuration to set up a sophisticated staged deployment, and it does not naturally support the notion of a "application monitoring period".

Most importantly, when depending on this deployment pipeline as the lifeline of your app or business, it's critical to have your deployment pipeline dashboarded succinctly and to have informative and useful troubleshooting information in the UI, which generic job servers cannot do.

Hacking on Immunity


Set your Mac up for development:



See the Procfile for the commands you can run to launch the web server, Resque, and clockwork.

To run them all together in one terminal, we use Foreman:

bundle exec foreman start

You can now access the web UI at http://localhost:3102. You'll need to already have mysql and redis-server already running.


The integration tests take a real, working webapp through the various stages of an Immunity workflow.

bundle exec rake test

Configuring your app in Immunity

Edit the config/immunity_apps.rb file to set up your app and its regions.

Then clone your app's repo into ~/immunity_repos, using the same directory name that you specified in config/immunity_apps.rb. Now, Immunity will poll this repo for new commits.