What does GitHub use to deploy? #38

tombell opened this Issue Sep 5, 2011 · 30 comments


None yet
8 participants

tombell commented Sep 5, 2011

I know GitHub uses Hubot to begin deploys, I was wondering what is actually used in the back end for deploying, is it just Capistrano, or do you guys use something else?


holman commented Sep 5, 2011

We use a few abstractions, but none of them are terribly complex:

  • You tell Hubot to deploy
  • Hubot talks to Heaven, which is a gem that handles all of the information needed to deploy all of our apps
  • Heaven then talks to Capistrano and does the actual commands needed to deploy your particular app

@holman holman closed this Sep 5, 2011

tombell commented Sep 11, 2011

Hey Zach,

Following on from holman#38. I'm from a .NET background where the deployment is simply point-click-wham-bam-thank-you-maam. I'm interested in getting a (smaller) similar set up to how you deploy at GitHub for a Ruby/Sinatra project I'm working on (and future projects) being able to deploy from IRC/Campfire.

Admittedly I have probably dived straight into the deep end instantly tinkering with nginx/unicorn/god/capistrano. I've pretty much got it working with manual deployment (i.e. running cap deploy from the project directory). I'm now interested in streamlining the process and eventually making it automatic.

I have a few questions about parts of the process:

Provisioning the server. I'm going to assume you guys are using chef to do this. Do you have any good resources for getting up to speed with chef.

Ruby version. Are you running REE or MRI 1.8/1.9.(2/3) and do you build a version of ruby during the provisioning of the server, or do you use a distribution package of Ruby? If you build a version of Ruby, do you install into a different prefix than the defaults. I'm currently using 1.9.3-preview1 build and installed into /opt/ruby1.9.3-preview1 and putting that into my users PATH. However I have to set the PATH variable to include this path in my capistrano deploy.rb. This feels like a dirty hack.

Finally, configuration files. Currently I have a config.yml which I have in config/config.yml but I exclude this from the git repository. As part of the cap deploy:setup I put this file into shared_path/config/config.yml and then symlink it during the actual deploy. So is there any better way for handling configuration files which are committed into the git repository?

Hopefully you'll be able to help kill my curiosity as to how things work at GitHub :)


holman commented Sep 12, 2011

Provisioning the server.

Actually, we use Puppet for production machines. As for chef, I found chef's docs to be pretty helpful, actually; their wiki area is pretty well-documented.

Ruby version.

We use a packaged (iirc) version of ree (unless it's a 1.9 app). Might want to ping @rodjek for more details if you'd like them, though.

Finally, configuration files.

This might be of some help here: https://github.com/blog/470-deployment-script-spring-cleaning

snelson commented Sep 23, 2011


I'm curious about the "Heaven" gem. Can you elaborate?


holman commented Sep 24, 2011

@snelson - Heaven is just a small Ruby wrapper around Capistrano. It's a simple Sinatra app that translates requests into cap commands- so developers can talk to Hubot in plain English and Heaven translates it into Capistrano commands. That way everyone doesn't need to know yet another syntax (and server commands and config).

snelson commented Sep 24, 2011

Nice ... do you keep all your deployment configs in Heaven then? So, individual apps aren't capified, but rather Heaven handles it all?


holman commented Sep 25, 2011

Yup! It knows where everything is and can handle config accordingly.

snelson commented Sep 27, 2011

Awesome man, I like it. Thanks for sharing!

minter commented Nov 16, 2011

Are you shelling out to call cap, or are you loading up the Capistrano configs in Ruby and manipulating them through the Capistrano API?

Any examples/pointers on how to do that?


holman commented Nov 17, 2011

We load the config files in Ruby, load them into Capistrano directly and invoke them that way.

minter commented Nov 18, 2011

Interesting. I've tried something similar, and it wasn't working. I was require'ing capistrano, then require'ing the deploy.rb, and got:

LoadError: Please require this file from within a Capistrano recipe

Do you have an example on how you would use capistrano programmatically and load up/execute deploy recipes? That would be a huge help.


holman commented Nov 18, 2011

Ah, looks like we just programmatically create the recipes and then shell out.

minter commented Nov 18, 2011

Gotcha - that makes more sense with what I was seeing. Thanks!

Any chance you'd open source the Heaven sinatra app?


holman commented Nov 27, 2012

Sure, although I think everyone who works on it are pretty busy with other things right now, so it's just not a priority at the moment.

Awh. I'm guessing I cannot bribe them with beers or hot pockets...

I guess I have a nice weekend project then. Thanks for the answer!

yevgenko commented Dec 3, 2012

I'm curious which type of versioning github use and where it reside in deployment flow, i.e. when exactly you do "bump version" and tagging?


holman commented Dec 3, 2012

We don't really tag or version our main GitHub app at all.

@rileytg rileytg referenced this issue in github/janky Dec 5, 2012


Deploy app after all tests pass #119

yevgenko commented Jan 7, 2013

Okay, here is my try to replicate Heaven, for now it really Simple web API in front of capistrano that translates requests into cap commands and let you manage multiple projects(recipes sets). Can be fairly easy integrated with Hubot, I think.

But the big plan is Safety First as described in Deploying at GitHub, which is another interesting thing that pertains to Heaven's magic.

How long do Github deployments normally take, both script running but also time including server restarts, qa time etc. I'm just simply amazed by the amount of deployments that can happen, they must be blazing fast 🔥


holman commented Oct 17, 2013

Deploys to production usually take somewhere around fifty to seventy seconds. Deploys to our staff-only environments take around twenty.

Wow! That's awesome, now is that primary one large app or many small apps? As well how long are Github's test suite(s)?

I'm really interested in doing a lot of deployment optimizations and have been trying to work out better processes and flows for the projects I'm working on. I've been running benchmarks against base Rails apps with out of the box Capistrano setup and they're like ~50-70 without any extra tasks.


holman commented Oct 17, 2013

primary one large app

One large app.

As well how long are Github's test suite(s)

Test suite has been a little sluggish the last few weeks: 80-100 seconds or so. Ideally we'd get them back down to 60-70 seconds.

😮 Awesome, thanks @holman!

Do you have any tips how you guys have scaled that many people working on a single codebase… other teams (assuming you work in teams) not clobbering 💥 other teams work, I've seen that to be a challenge. Is it just a general clear vision 👀 for where the whole app is moving?

I appreciate really you're input 👍 by the way and find all your talks very inspiring, as well neat way of creating an FAQ kinda thing with PRs never thought of them like that.


holman commented Oct 17, 2013

Is it just a general clear vision 👀 for where the whole app is moving?

Yup, for the most part. Teams split and work on their own stuff, but we try to have a pretty cohesive vision of what we want overall.

Do you have any tips how you guys have scaled that many people working on a single codebase

That's a pretty big question, hah. :) I cover some of it in my talks… that's probably a good start (although sounds like you've seen a few already!)

Yeah I have 😄, and yeah was kind of vague sorry, your Scaling Github talk was helpful (really pushed the use of PRs) I guess the biggest problem I've seen/trying to work through is getting everything to that "done" state, where teams don't start relying or building off other pull requests causing a backlog of pull requests that eventually get stale if one is being held up for an extend period of time. Does that make sense? Have you seen that?


holman commented Oct 17, 2013

where teams don't start relying or building off other pull requests causing a backlog of pull requests that eventually get stale if one is being held up for an extend period of time.

Very rarely do we have branches branched off branches. That helps us avoid that specific issue you mentioned: you're not tied to waiting on someone else.

A specific example: right now I'm working on something that hopefully will be shipped after someone else ships a related feature. i.e., I want their branch in my branch. But rather than do that, we both have feature flags in our code so that we could ship both independently of each other. It's a little more overhead, but it's pretty nice knowing that I'm not going to be a blocker on your work. This way we can both ship to the master branch when we're ready, and then ship to the general population with each of our features are done.

Perfect that's exactly where we're at, it's just a hard cultural change for people aside from the overhead but it all makes sense.

Thanks again! Definitely owe you a 🍺 or 🍻

swmcc commented Oct 17, 2013

👍 this thread :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment