Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Merge pull request #5 from dreverri/refactor

Refactor
  • Loading branch information...
commit cd4ae5cda6e1f4e5b1cd702f2e1ee3164206d465 2 parents 64d399c + c949950
@dreverri authored
View
3  .gitignore
@@ -1 +1,2 @@
-config.yml
+config/projects.yml
+.DS_Store
View
1  .rvmrc
@@ -0,0 +1 @@
+rvm use ruby-1.9.2@github_post_receive --create
View
11 Gemfile
@@ -0,0 +1,11 @@
+source :rubygems
+
+gem 'sinatra'
+gem 'grit'
+gem 'posix-spawn'
+
+group :development do
+ gem 'rake'
+ gem 'rack-test'
+ gem 'rspec'
+end
View
36 Gemfile.lock
@@ -0,0 +1,36 @@
+GEM
+ remote: http://rubygems.org/
+ specs:
+ diff-lcs (1.1.2)
+ grit (2.4.1)
+ diff-lcs (~> 1.1)
+ mime-types (~> 1.15)
+ mime-types (1.16)
+ posix-spawn (0.3.6)
+ rack (1.3.0)
+ rack-test (0.6.0)
+ rack (>= 1.0)
+ rake (0.9.2)
+ rspec (2.6.0)
+ rspec-core (~> 2.6.0)
+ rspec-expectations (~> 2.6.0)
+ rspec-mocks (~> 2.6.0)
+ rspec-core (2.6.3)
+ rspec-expectations (2.6.0)
+ diff-lcs (~> 1.1.2)
+ rspec-mocks (2.6.0)
+ sinatra (1.2.6)
+ rack (~> 1.1)
+ tilt (< 2.0, >= 1.2.2)
+ tilt (1.3.2)
+
+PLATFORMS
+ ruby
+
+DEPENDENCIES
+ grit
+ posix-spawn
+ rack-test
+ rake
+ rspec
+ sinatra
View
86 README.md
@@ -0,0 +1,86 @@
+# Description
+
+Sinatra app that handles HTTP POSTs from GitHub's post-receive
+webhook.
+
+# Usage
+
+* Create the file `config/projects.yml`
+
+* Add a project to the projects file
+
+```yaml
+path/to/deploy:
+ name: repo-sync-webhook
+ branch: master
+ cmd: rake
+```
+
+* Use the `rackup` command to start the app on port `9292` bound to
+ `0.0.0.0`
+
+* Setup a [post receive
+ hook](http://help.github.com/post-receive-hooks/) in the Github
+ admin panel
+
+# Config
+
+Project definitions are specified in a `projects.yml` file.
+
+# Projects
+
+The config file should contain a hash of projects. The key of the hash
+is used as the deploy path. Each project should define the following
+parameters:
+
+* name - Name of the repository
+* branch - Name of the branch to respond to
+* token - Token to be matched to incoming requests (optional)
+* cmd - Command to run when the post commit hook is received
+
+Each defined project should be specific to a particular repository and
+branch. When the incoming request matches the defined repository,
+branch, and token the cmd will be executed.
+
+```yaml
+path/to/deploy:
+ name: repo-sync-webhook
+ branch: master
+ token: secret
+ cmd: rake
+```
+
+# Deploy Path
+
+The path defined for a project (e.g. `path/to/deploy`) stores the currently
+processed commit. Each post receive hook creates a new directory
+within the deploy path using the id of the received commit.
+
+A symlink (`current`) is maintained which always points to the most
+recently processed commit.
+
+# Token
+
+Projects may specify a token parameter. Incoming requests will be
+expected to have a token query parameter with a value that matches the
+value defined in the config file.
+
+If a token is defined for a project the post commit URL should be
+similar to `http://hostname:9292/notify?token=project_token`
+
+# Process
+
+When a post receive hook is received this app will do the following:
+
+* Read repository name, branch name, and commit id from payload
+* Look for matching projects in the config file
+* Clone the repo to a new directory and checkout the commit
+* Set the current working directory to this new directory
+* Run the project's defined cmd
+ * On success - update current symlink and remove old version
+ * On failure - log the failure and destroy the commit directory
+
+# To Do
+
+* Use mutex per project rather than Sinatra's global mutex
+* Package as a gem
View
74 README.org
@@ -1,74 +0,0 @@
-* Description
-
- Sinatra app that handles HTTP POSTs from GitHub's post-receive
- webhook.
-
-* Usage
-
- The following command will start a server on port 4567 bound to "0.0.0.0".
-
-#+BEGIN_SRC bash
- ruby sync.rb
-#+END_SRC
-
-* Config
-
- This app supports project definitions specified in a "config.yml"
- file. A [[./config.example.yml][sample]] is provided in this repository.
-
-* Projects
-
- The config file should contain a list of projects. Each project should
- define the following parameters:
-
- - name :: Name of the repository
- - branch :: Name of the branch to respond to
- - token :: Token to be matched to incoming requests (optional)
- - root :: The repository will be cloned to this directory
- - cmd :: Command to run when post commit hook is received
-
- The assumption is that each defined project should be specific to a
- particular repository and branch. When the incoming request matches
- the defined repository, branch, and token the cmd will be executed.
-
-* Token
-
- Projects may specify a token parameter. Incoming requests will be
- expected to have a token query parameter with a value that matches
- the value defined in the config file.
-
- If a token is defined for a project the post commit URL should be
- similar to "http://hostname:4567/notify?token=project_token"
-
-* Process
-
- When a post receive hook is received this app will do the following:
-
- - Read repository name, branch name, and commit id from payload
- - Look for matching projects
- - Mirror repository or fetch updates
- #+BEGIN_SRC bash
- # Mirror repository
- git clone --mirror #{repository} #{cache}
- #+END_SRC
- #+BEGIN_SRC bash
- # Fetch updates
- git --git-dir=#{cache} fetch
- #+END_SRC
- - Checkout the commit
- #+BEGIN_SRC
- git clone #{cache} #{commit_path}
- git --git-dir=#{commit_path}/.git --work-tree=#{commit_path} \
- checkout -f #{commit_id}
- #+END_SRC
- - Change directory to the repository directory
- - Run the cmd defined in the config
-
- The cache and commit_path are directories created in the root
- directory defined in the config file.
-
-* To Do
-
- - Use mutex per project rather than Sinatra's global mutex
- (set :lock, true)
- - Package as a gem
View
15 Rakefile
@@ -0,0 +1,15 @@
+require 'rubygems'
+require 'bundler'
+
+Bundler.setup
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+
+desc "Run Unit Specs Only"
+RSpec::Core::RakeTask.new(:spec) do |spec|
+ spec.pattern = "spec/github_post_receive/**/*_spec.rb"
+ spec.rspec_opts = ["--color", "--format", "doc"]
+end
+
+task :default => :spec
View
6 config.example.yml
@@ -1,6 +0,0 @@
-:projects:
- - :name: riak_wiki
- :branch: master
- :token: project_token
- :root: .
- :cmd: gollum-site generate && (PREV=`readlink ../current`.. || PREV="") && rm -f ../current && ln -s `pwd`/_site ../current && ([ `pwd` -ef `cd $PREV && pwd` ] || rm -rf $PREV)
View
12 config.ru
@@ -0,0 +1,12 @@
+require 'rubygems'
+require 'bundler'
+
+Bundler.setup
+
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), 'lib'))
+require 'github_post_receive'
+GithubPostReceive::App.load_config("config/projects.yml")
+if File.exists?("config/logger.yml")
+ GithubPostReceive::App.load_logger_config("config/logger.yml")
+end
+run GithubPostReceive::App
View
1  config/logger.yml
@@ -0,0 +1 @@
+level: ERROR
View
5 config/projects.example.yml
@@ -0,0 +1,5 @@
+wiki:
+ name: riak_wiki
+ branch: master
+ token: project_token
+ cmd: gollum-site generate
View
9 lib/github_post_receive.rb
@@ -0,0 +1,9 @@
+require 'json'
+require 'yaml'
+require 'grit'
+require 'posix-spawn'
+require 'sinatra/base'
+require 'github_post_receive/payload'
+require 'github_post_receive/project'
+require 'github_post_receive/deployment'
+require 'github_post_receive/app'
View
66 lib/github_post_receive/app.rb
@@ -0,0 +1,66 @@
+module GithubPostReceive
+ class App < Sinatra::Application
+ def self.load_config(fname)
+ hsh = YAML.load(ERB.new(File.read(File.expand_path(fname))).result)
+ load_hash(hsh)
+ end
+
+ def self.load_hash(hsh)
+ set :projects, hsh.map { |path, config| Project.new(path, config) }
+ end
+
+ def self.load_logger_config(fname)
+ hsh = YAML.load(ERB.new(File.read(File.expand_path(fname))).result)
+ self.load_logger(hsh)
+ end
+
+ def self.load_logger(hsh={})
+ $logger.close if $logger
+ $logger = ::Logger.new(hsh['device'] || STDOUT)
+ $logger.level = ::Logger.const_get(hsh['level'] || ENV['LOGGER_LEVEL'] || 'ERROR')
+ $logger.datetime_format = hsh['datetime_format'] || "%Y-%m-%d %H:%M:%S"
+ Grit.logger = $logger
+ Grit.debug = $logger.debug?
+ $logger
+ end
+
+ def self.logger
+ $logger ||= load_logger
+ end
+
+ configure do
+ enable :lock
+ end
+
+ get '/' do
+ "Nothing to see here"
+ end
+
+ post '/notify' do
+ process_request(params, settings.projects)
+ "Thank you"
+ end
+
+ helpers do
+ def process_request(params, projects)
+ payload = Payload.from_params(params)
+ projects.each do |project|
+ if project.match?(payload)
+ async = (params[:async] == "true")
+ project.deploy(payload.url, payload.commit_id, async)
+ end
+ end
+ rescue GithubPostReceive::AlreadyDeployed => e
+ logger.error("Received notification for an already deployed commit: #{e.message}")
+ end
+
+ def url(payload, project)
+ project['remote'] || payload['repository']['url']
+ end
+
+ def logger
+ self.class.logger
+ end
+ end
+ end
+end
View
76 lib/github_post_receive/deployment.rb
@@ -0,0 +1,76 @@
+module GithubPostReceive
+ class AlreadyDeployed < StandardError; end
+ class Deployment
+ include POSIX::Spawn
+
+ attr_reader :project, :remote, :commit_id, :repo
+
+ def initialize(project, remote, commit_id)
+ App.logger.debug "Initializing deployment for commit #{commit_id}"
+ @project = project
+ @remote = remote
+ @commit_id = commit_id
+ init_cache
+ init_repo
+ end
+
+ def init_cache
+ cache_path = File.join(@project.path, 'cache.git')
+ @cache = Grit::Repo.init_bare(cache_path)
+ end
+
+ def init_repo
+ path = File.join(@project.path, @commit_id)
+ raise AlreadyDeployed.new(path) if File.exists? path
+ @repo = Grit::Repo.init(path)
+ end
+
+ def git_options
+ {:raise => true, :timeout => @project.timeout}
+ end
+
+ def list_mirrors(repo)
+ Dir.glob(File.join(repo.path, 'refs/heads/*')).map do |path|
+ File.basename(path)
+ end
+ end
+
+ def cache
+ if list_mirrors(@cache).empty?
+ App.logger.debug "Adding #{@remote} as 'origin'"
+ @cache.git.remote(git_options, 'add', '--mirror', 'origin', @remote)
+ end
+ App.logger.debug "Updating from #{@remote}"
+ @cache.git.remote(git_options, 'update')
+ end
+
+ def clone
+ App.logger.debug "Adding #{@cache.path} as 'origin'"
+ @repo.git.remote(git_options, 'add', 'origin', @cache.path)
+ App.logger.debug "Fetching #{@cache.path}"
+ @repo.git.fetch(git_options, 'origin')
+ end
+
+ def checkout
+ options = git_options.merge({:base => false, :chdir => @repo.working_dir})
+ @repo.git.checkout(options, @commit_id)
+ end
+
+ def run
+ return true if (@project.cmd.nil? || @project.cmd.empty?)
+ process = Child.new(@project.cmd, :chdir => @repo.working_dir)
+ raise process.err unless process.status.success?
+ return true
+ end
+
+ def deploy
+ cache
+ clone
+ checkout
+ run
+ rescue => e
+ App.logger.error "Deploy failed [#{@project.name}][#{@project.path}][#{@commit_id}]: #{e.inspect}"
+ return false
+ end
+ end
+end
View
29 lib/github_post_receive/payload.rb
@@ -0,0 +1,29 @@
+module GithubPostReceive
+ class Payload
+ attr_reader :name, :branch, :commit_id, :url, :token
+
+ def self.from_params(params)
+ payload = JSON.parse(params[:payload])
+ if payload['repository']['private']
+ owner_name = payload['repository']['owner']['name']
+ repo_name = payload['repository']['name']
+ url = "git@github.com:/#{owner_name}/#{repo_name}.git"
+ else
+ url = payload['repository']['url']
+ end
+ new(payload['repository']['name'],
+ payload['ref'].split('/').last,
+ payload['after'],
+ url,
+ params[:token])
+ end
+
+ def initialize(name, branch, commit_id, url, token=nil)
+ @name = name
+ @branch = branch
+ @commit_id = commit_id
+ @url = url
+ @token = token
+ end
+ end
+end
View
54 lib/github_post_receive/project.rb
@@ -0,0 +1,54 @@
+module GithubPostReceive
+ class Project
+ attr_accessor :path, :name, :branch, :cmd, :token, :timeout
+
+ def initialize(path, options = {})
+ @path = path
+ @name = options['name']
+ @branch = options['branch']
+ @cmd = options['cmd']
+ @token = options['token']
+ @timeout = options['timeout'] || false
+ end
+
+ def match?(payload)
+ name == payload.name &&
+ branch == payload.branch &&
+ (token.nil? || token == payload.token)
+ end
+
+ def link_path
+ File.join(@path, 'current')
+ end
+
+ def deploy(remote, commit_id, async=false)
+ # Prepare deployment before starting work in the background in
+ # order to prevent a race condition between multiple
+ # deployments. This assumes the starting process (e.g. a Sinatra
+ # application) maintains a lock per project directory
+ deployment = prepare(remote, commit_id)
+ thread = Thread.new { really_deploy(deployment) }
+ thread.join unless async
+ end
+
+ def prepare(remote, commit_id)
+ Deployment.new(self, remote, commit_id)
+ end
+
+ def really_deploy(deployment)
+ new_path = deployment.repo.working_dir
+ if deployment.deploy
+ if File.symlink?(link_path)
+ old_path = File.readlink(link_path)
+ File.unlink(link_path)
+ File.symlink(new_path, link_path)
+ FileUtils.rm_rf(old_path)
+ else
+ File.symlink(new_path, link_path)
+ end
+ else
+ FileUtils.rm_rf(new_path)
+ end
+ end
+ end
+end
View
118 spec/github_post_receive/app_spec.rb
@@ -0,0 +1,118 @@
+require File.expand_path("../spec_helper", File.dirname(__FILE__))
+require 'grit'
+
+describe "application" do
+ include Rack::Test::Methods
+
+ def app
+ GithubPostReceive::App
+ end
+
+ before do
+ @repo = Grit::Repo.init(File.join(Dir.mktmpdir))
+
+ i = @repo.index
+ i.add('foo', 'foo')
+ @c1 = i.commit("add foo")
+ i.add('bar', 'bar')
+ @c2 = i.commit("add bar", [@c1])
+
+ @path = Dir.mktmpdir
+ @hsh = {
+ @path => {
+ "name" => "baz",
+ "branch" => "master",
+ "cmd" => "touch new.txt"
+ }
+ }
+
+ GithubPostReceive::App.load_hash(@hsh)
+
+ @payload = {
+ "repository" => {
+ "name" => "baz",
+ "url" => @repo.path
+ },
+ "ref" => "master",
+ "after" => @c1
+ }
+ @payload2 = @payload.dup
+ @payload2['after'] = @c2
+
+ post '/notify', {:payload => @payload.to_json}
+ end
+
+ after do
+ FileUtils.rm_rf(@repo.working_dir)
+ FileUtils.rm_rf(@path)
+ end
+
+ it "should clone repository" do
+ Dir.exists?(File.join(@path, @c1)).should be_true
+ end
+
+ it "should run the project cmd" do
+ File.exists?(File.join(@path, @c1, 'new.txt')).should be_true
+ end
+
+ describe "aync process" do
+ before do
+ GithubPostReceive::App.projects.first.cmd = "sleep 1; touch other.txt"
+ post '/notify?async=true', {:payload => @payload2.to_json}
+ end
+
+ it "should return immediately and deploy asynchronously" do
+ File.exists?(File.join(@path, @c2, 'other.txt')).should_not be_true
+ last_response.should be_ok
+ sleep(2)
+ File.exists?(File.join(@path, @c2, 'other.txt')).should be_true
+ end
+ end
+
+ describe "double posts" do
+ before do
+ GithubPostReceive::App.projects.first.cmd = "touch other.txt"
+ post '/notify', {:payload => @payload.to_json}
+ end
+
+ it "should not re-run already deployed commits" do
+ File.exists?(File.join(@path, @c1, 'new.txt')).should be_true
+ File.exists?(File.join(@path, @c1, 'other.txt')).should_not be_true
+ end
+ end
+
+ describe "when deployed multiple times" do
+ describe "and cmd passes" do
+ before do
+ post '/notify', {:payload => @payload2.to_json}
+ end
+
+ it "should update the current symlink" do
+ File.readlink(File.join(@path, 'current')).should == File.join(@path, @c2)
+ end
+
+ it "should remove the old version" do
+ Dir.exists?(File.join(@path, @c1)).should_not be_true
+ end
+ end
+
+ describe "and cmd fails" do
+ before do
+ GithubPostReceive::App.projects.first.cmd = "false"
+ post '/notify', {:payload => @payload2.to_json}
+ end
+
+ it "should not change symlink" do
+ File.readlink(File.join(@path, 'current')).should == File.join(@path, @c1)
+ end
+
+ it "should not remove old version" do
+ Dir.exists?(File.join(@path, @c1)).should be_true
+ end
+
+ it "should remove the new version" do
+ Dir.exists?(File.join(@path, @c2)).should_not be_true
+ end
+ end
+ end
+end
View
75 spec/github_post_receive/project_spec.rb
@@ -0,0 +1,75 @@
+require File.expand_path("../spec_helper", File.dirname(__FILE__))
+
+describe "project deploy" do
+ before do
+ @repo = Grit::Repo.init(File.join(Dir.mktmpdir))
+
+ @i = @repo.index
+ @i.add('foo', 'foo')
+ @c1 = @i.commit("add foo")
+
+ cmd = 'touch new.txt'
+ options = {'branch' => 'master', 'cmd' => cmd}
+ @project = GithubPostReceive::Project.new(Dir.mktmpdir, options)
+ @project.deploy(@repo.path, @c1)
+ end
+
+ after do
+ FileUtils.rm_rf(@repo.working_dir)
+ FileUtils.rm_rf(@project.path)
+ end
+
+ it "should cache the repository" do
+ cache_path = File.join(@project.path, 'cache.git')
+ Dir.exists?(cache_path).should be_true
+ @repo.git.rev_parse({:base => false, :chdir => cache_path}, "HEAD").should == @c1
+ end
+
+ it "should clone the repository" do
+ Dir.exists?(File.join(@project.path, @c1, '.git')).should be_true
+ end
+
+ it "should run the specified command" do
+ File.exists?(File.join(@project.path, @c1, 'new.txt')).should be_true
+ end
+
+ it "should setup a symlink" do
+ current = File.join(@project.path, 'current')
+ File.readlink(current).should == File.join(@project.path, @c1)
+ Dir[File.join(current, "*")].map { |d| File.basename(d) }.sort.should ==
+ ['foo', 'new.txt']
+ end
+
+ it "should raise an exception if already deployed and maintain current symlink" do
+ lambda { @project.deploy(@repo.path, @c1) }.
+ should raise_error GithubPostReceive::AlreadyDeployed
+ current = File.join(@project.path, 'current')
+ Dir.exists?(File.join(current, '.git')).should be_true
+ File.exists?(File.join(current, 'new.txt')).should be_true
+ end
+
+ describe "project update" do
+ before do
+ @i.add('bar', 'bar')
+ @c2 = @i.commit("add bar", [@c1])
+
+ @project.deploy(@repo.path, @c2)
+ end
+
+ it "should update the cache repository" do
+ cache_path = File.join(@project.path, 'cache.git')
+ @repo.git.rev_parse({:base => false, :chdir => cache_path}, "HEAD").should == @c2
+ end
+
+ it "should replace the old symlink" do
+ current = File.join(@project.path, 'current')
+ File.readlink(current).should == File.join(@project.path, @c2)
+ Dir[File.join(current, "*")].map { |d| File.basename(d) }.sort.should ==
+ ['bar', 'foo', 'new.txt']
+ end
+
+ it "should remove the old commit directory" do
+ Dir.exists?(File.join(@project.path, @c1)).should_not be_true
+ end
+ end
+end
View
18 spec/spec_helper.rb
@@ -0,0 +1,18 @@
+require 'rubygems'
+require 'bundler'
+
+Bundler.setup
+
+require 'rspec'
+require 'rack/test'
+
+# Set's the appropriate server settings for Ripple
+ENV['RACK_ENV'] = 'test'
+
+# Application
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '../lib'))
+require 'github_post_receive'
+
+# Test helpers
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'support/rspec'
View
9 spec/support/rspec.rb
@@ -0,0 +1,9 @@
+RSpec.configure do |config|
+ config.mock_with :rspec
+
+ config.after(:each) do
+ end
+
+ config.filter_run :focus => true
+ config.run_all_when_everything_filtered = true
+end
View
88 sync.rb
@@ -1,88 +0,0 @@
-require 'rubygems'
-require 'sinatra'
-require 'json'
-require 'yaml'
-
-# TODO: Make config file configurable
-CONFIG = YAML::load_file("config.yml") unless defined? CONFIG
-
-set :lock, true
-
-get '/' do
- 'Nothing to see here'
-end
-
-post '/notify' do
- process_request(params, CONFIG)
- "Thank you"
-end
-
-def process_request(params, config)
- payload = JSON.parse(params[:payload])
- name = payload['repository']['name']
- branch = payload["ref"].split("/").last
- commit_id = payload['after']
-
- config[:projects].each do |project|
- if project[:name] == name && project[:branch] == branch
- if project[:token].nil? || project[:token] == params[:token]
- puts "Processing #{name}:#{branch}"
- root = project[:root]
- cmd = project[:cmd]
- remote = url(payload, project)
-
- process_project(root, name, commit_id, remote, cmd)
- else
- puts "The provided token, #{params[:token]}, did not match"
- end
- end
- end
-end
-
-def process_project(root, name, commit_id, remote, cmd)
- repo_path = File.join(root, name)
- cache_path = File.join(repo_path, "cache")
- commit_path = File.join(repo_path, commit_id)
-
- # Mirror repo or fetch updates
- if File.exists?cache_path
- puts "Fetching updates to #{cache_path}"
- fetch(cache_path)
- else
- puts "Mirroring repository #{remote} to #{cache_path}"
- mirror(remote, cache_path)
- end
-
- # Check out commit
- # What is the least surprising behavior when the commit path already exists?
- unless File.exists?commit_path
- puts "Checking out #{commit_id}"
- checkout(cache_path, commit_path, commit_id)
-
- # Change to commit directory and run cmd
- puts "Running #{cmd}"
- %x[cd #{commit_path} && #{cmd}]
- else
- puts "The directory for this commit already exists: #{commit_path}"
- end
-end
-
-def mirror(repo, cache)
- %x[git clone --bare #{repo} #{cache} && (cd #{cache} && git remote add --mirror origin #{repo})]
-end
-
-def fetch(cache)
- %x[git --git-dir=#{cache} fetch]
-end
-
-def checkout(cache, commit_path, commit_id)
- clone = "git clone #{cache} #{commit_path}"
- checkout_opts = "--git-dir=#{commit_path}/.git --work-tree=#{commit_path}"
- checkout = "git #{checkout_opts} checkout -f #{commit_id}"
- %x[#{clone} && #{checkout}]
-end
-
-def url(payload, project)
- url = payload['repository']['url']
- project[:git_url] || url.gsub(/https:\/\//, 'git://') + '.git'
-end
Please sign in to comment.
Something went wrong with that request. Please try again.