Ruby HTML CSS CoffeeScript Shell JavaScript
Switch branches/tags
Clone or download
Permalink
Failed to load latest commit information.
app Stop storing hook deliveries in the database Aug 2, 2018
bin Workaround annoying sqlite3 adapter bug in bin/bootstrap Feb 17, 2017
config Expose env vars to allow to create git tags Jun 18, 2018
db/migrate Handle unregistered git authors Aug 2, 2018
docs Update permission list Jul 31, 2018
examples Add an example shipit.yml [fix #463] May 7, 2016
lib Set `ROLLBACK=1` for rollback commands Aug 14, 2018
log rails new Feb 26, 2014
script Skip seed testing for sqlite3 Dec 6, 2016
test Set `ROLLBACK=1` for rollback commands Aug 14, 2018
vendor/assets Handle cmd + f from inside text inputs Apr 18, 2016
.gitignore Ignore vim swap files Jun 25, 2018
.rubocop.yml Open source the server side code of our browser extension May 31, 2018
.rubocop_todo.yml Use safe navigation operator now that we require 2.3 Apr 24, 2018
.travis.yml Updare rubocop to target MRI 2.3 Apr 24, 2018
CHANGELOG.md Set `ROLLBACK=1` for rollback commands Aug 14, 2018
Gemfile Upgrade to Rubocop 0.52, update config accordingly, and whitelist som… Feb 27, 2018
Gemfile.lock Bump sprockets version due to CVE Jun 26, 2018
LICENSE.txt Update the year in LICENSE.txt Sep 14, 2015
README.md Add `title` property to tasks to control the task will be displayed i… May 28, 2018
Rakefile Ignore tests in dummy app May 21, 2015
dev.yml Updare rubocop to target MRI 2.3 Apr 24, 2018
railgun.yml Move dev/railgun to myshopify.io domain Sep 21, 2017
shipit-engine.gemspec Update omniauth-github [fix #792] May 22, 2018
template.rb Update template.rb May 22, 2018

README.md

Shipit - Documentation

Build Status Gem Version

Shipit is a deployment tool that makes shipping code better for everyone. It's especially great for large teams of developers and designers who work together to build and deploy GitHub repos. You can use it to:

  • add new applications to your deployment environment without having to change core configuration files repeatedly — shipit.yml is basically plug and play
  • control the pace of development by pushing, locking, and rolling back deploys from within Shipit
  • enforce checklists and provide monitoring right at the point of deployment.

Shipit is compatible with just about anything that you can deploy using a script. It natively detects stacks using bundler and Capistrano, and it has tools that make it easy to deploy to Heroku or RubyGems. At Shopify, we've used Shipit to synchronize and deploy hundreds of projects across dozens of teams, using Python, Rails, RubyGems, Java, and Go.

This guide aims to help you set up, use, and understand Shipit.

Shipit requires a database (MySQL, PostgreSQL or SQLite3), redis, and Ruby 2.1 or superior.


Table of contents

I. INSTALLATION & SETUP

II. USING SHIPIT

III. REFERENCE


I. INSTALLATION & SETUP

Installation

To create a new Shipit installation you can follow the setup guide.

Updating an existing installation

  1. If you locked the gem to a specific version in your Gemfile, update it there.
  2. Update the shipit-engine gem with bundle update shipit-engine.
  3. Install new migrations with rake shipit:install:migrations db:migrate.

Specific updates requiring more steps

If you are upgrading from 0.21 or older, you will have to update the configuration. Please follow the dedicated upgrade guide


II. USING SHIPIT

The main workflows in Shipit are adding stacks, working on stacks, and configuring stacks.

A stack is composed of a GitHub repository, a branch, and a deployment environment. Shipit tracks the commits made to the branch, and then displays them in the stack overview. From there, you can deploy the branch to whatever environment you've chosen (some typical environments include production, staging, performance, etc.).

Add a new stack

  1. From the main page in Shipit, click Add a stack.
  2. On the Create a stack page, enter the required information:
    • Repo
    • Branch
    • Environment
    • Deploy URL
  3. When you're finished, click Create stack.

Work on an existing stack

  1. If you want to browse the list of available stacks, click Show all stacks on the main page in Shipit. If you know the name of the stack you're looking for, enter it in the search field.
  2. Click the name of the stack you want to open.
  3. From a stack's overview page, you can:
    • review previous deploys
    • deploy any undeployed commits by clicking Deploy
    • rollback to an earlier build by clicking Rollback to this deploy
    • adjust the stack's settings by clicking the gear icon in the page header
    • perform any custom tasks that are defined in the shipit.yml file
  4. When you're ready to deploy an undeployed commit, click the relevant Deploy button on the stack's overview page.
  5. From the Deploy page, complete the checklist, then click Create deploy.

Edit stack settings

To edit a stack's settings, open the stack in Shipit, then click the gear icon in the page header.

From a stack's Settings page, you can:

  • change the deploy URL
  • enable and disable continuous deployment
  • lock and unlock deploys through Shipit
  • resynchronize the stack with GitHub
  • delete the stack from Shipit

III. REFERENCE

Configuring shipit.yml

The settings in the shipit.yml file relate to the different things you can do with Shipit:

All the settings in shipit.yml are optional. Most applications can be deployed from Shipit without any configuration.

Also, if your repository is deployed different ways depending on the environment, you can have an alternative shipit.yml by including the environment name.

For example for a stack like: my-org/my-repo/staging, shipit.staging.yml will have priority over shipit.yml.


Installing dependencies

The dependencies step allows you to install all the packages your deploy script needs.

Bundler

If your application uses Bundler, Shipit will detect it automatically and take care of the bundle install and prefix your commands with bundle exec.

By default, the following gem groups will be ignored:

  • default
  • production
  • development
  • test
  • staging
  • benchmark
  • debug

The gems you need in order to deploy should be in a different group, such as deploy.

For example:

dependencies:
  bundler:
    without:
      - development
      - test
      - debug

Other dependencies

If your deploy script uses another tool to install dependencies, you can install them manually via dependencies.override:

dependencies:
  override:
    - npm install

dependencies.pre If you wish to execute commands before Shipit installs the dependencies, you can specify them here.

For example:

dependencies:
  pre:
    - mkdir tmp/
    - cp -R /var/cache/ tmp/cache

dependencies.post If you wish to execute commands after Shipit installed the dependencies, you can specify them here:

For example:

dependencies:
  post:
    - cp -R tmp/cache /var/cache/

Deployment

The deploy and rollback sections are the core of Shipit:

deploy.override contains an array of the shell commands required to deploy the application. Shipit will try to infer it from the repository structure, but you can change the default inference.

For example:

deploy:
  override:
    - ./script/deploy

deploy.pre If you wish to execute commands before Shipit executes your deploy script, you can specify them here.

For example:

deploy:
  pre:
    - ./script/notify_deploy_start

deploy.post If you wish to execute commands after Shipit executed your deploy script, you can specify them here.

For example:

deploy:
  post:
    - ./script/notify_deploy_end

You can also accept custom environment variables defined by the user that triggers the deploy:

deploy.variables contains an array of variable definitions.

For example:

deploy:
  variables:
    -
      name: RUN_MIGRATIONS
      title: Run database migrations on deploy
      default: 1

deploy.variables.select will turn the input into a <select> of values.

For example:

deploy:
  variables:
    -
      name: REGION
      title: Run a deploy in a given region
      select:
        - east
        - west
        - north

deploy.max_commits defines the maximum number of commits that should be shipped per deploys. Defaults to 8.

Human users will be warned that they are not respecting the recommendation, but allowed to continue. However continuous delivery will respect this limit. If there is no deployable commits in this range, a human intervention will be required.

For example:

deploy:
  max_commits: 5

deploy.interval defines the interval between the end of a deploy and the next deploy, when continuous delivery is enabled. You can use s, m, h, d as units for seconds, minutes, hours, and days. Defaults to 0, which means a new deploy will start as soon as the current one finishes.

For example, this will wait 5 minutes after the end of a deploy before starting a new one:

deploy:
  interval: 5m

rollback.override contains an array of the shell commands required to rollback the application to a previous state. Shipit will try to infer it from the repository structure, but you can change the default inference. This key defaults to disabled unless Capistrano is detected.

For example:

rollback:
  override:
    - ./script/rollback

rollback.pre If you wish to execute commands before Shipit executes your rollback script, you can specify them here:

For example:

rollback:
  pre:
    - ./script/notify_rollback_start

rollback.post If you wish to execute commands after Shipit executed your rollback script, you can specify them here:

For example:

rollback:
  post:
    - ./script/notify_rollback_end

fetch contains an array of the shell commands that Shipit executes to check the revision of the currently-deployed version. This key defaults to disabled.

For example:

fetch:
  curl --silent https://app.example.com/services/ping/version

Kubernetes

kubernetes allows to specify a Kubernetes namespace and context to deploy to.

For example:

kubernetes:
  namespace: my-app-production
  context: tier4

kubernetes.template_dir allows to specify a Kubernetes template directory. It defaults to ./config/deploy/$ENVIRONMENT

Environment

machine.environment contains the extra environment variables that you want to provide during task execution.

For example:

machine:
  environment:
    key: val # things added as environment variables

Directory

machine.directory specifies a subfolder in which to execute all tasks. Useful for repositories containing multiple applications or if you don't want your deploy scripts to be located at the root.

For example:

machine:
  directory: scripts/deploy/

Cleanup

machine.cleanup specifies whether or not the deploy working directory should be cleaned up once the deploy completed. Defaults to true, but can be useful to disable temporarily to investigate bugs.

For example:

machine:
  cleanup: false

CI

ci.require contains an array of the statuses context you want Shipit to disallow deploys if any of them is missing on the commit being deployed.

For example:

ci:
  require:
    - ci/circleci

ci.hide contains an array of the statuses context you want Shipit to ignore.

For example:

ci:
  hide:
    - ci/circleci

ci.allow_failures contains an array of the statuses context you want to be visible but not to required for deploy.

For example:

ci:
  allow_failures:
    - ci/circleci

ci.blocking contains an array of the statuses context you want to disallow deploys if any of them is missing or failing on any of the commits being deployed.

For example:

ci:
  blocking:
    - soc/compliance

Merge Queue

The merge queue allows developers to register pull requests which will be merged by Shipit once the stack is clear (no lock, no failing CI, no backlog). It can be enabled on a per stack basis via the settings page.

It can be customized via several shipit.yml properties:

merge.revalidate_after a duration after which pull requests that couldn't be merged are rejected from the queue. Defaults to unlimited.

For example:

merge:
  revalidate_after: 12m30s

merge.require contains an array of the statuses context that you want Shipit to consider as failing if they aren't present on the pull request. Defaults to ci.require if present, or empty otherwise.

For example:

merge:
  require:
    - continuous-integration/travis-ci/push

merge.ignore contains an array of the statuses context that you want Shipit not to consider when merging pull requests. Defaults to the union of ci.allow_failures and ci.hide if any is present or empty otherwise.

For example:

merge:
  ignore:
    - codeclimate

merge.method the merge method to use for this stack. If it's not set the default merge method will be used. Can be either merge, squash or rebase.

For example:

merge:
  method: squash

merge.max_divergence.commits the maximum number of commits a pull request can be behind its merge base, after which pull requests are rejected from the merge queue.

For example:

merge:
  max_divergence:
    commits: 50

merge.max_divergence.age a duration after the commit date of the merge base, after which pull requests will be rejected from the merge queue.

For example:

merge:
  max_divergence:
    age: 72h

Custom tasks

You can create custom tasks that users execute directly from a stack's overview page in Shipit. To create a new custom task, specify its parameters in the tasks section of the shipit.yml file. For example:

tasks.restart restarts the application.

tasks:
  restart:
    action: "Restart Application"
    description: "Sometimes needed if you want the application to restart but don't want to ship any new code."
    steps:
      - ssh deploy@myserver.example.com 'touch myapp/restart.txt'

By default, custom tasks are not allowed to be triggered while a deploy is running. But if it's safe for that specific task, you can change that behavior with the allow_concurrency attribute:

tasks:
  flush_cache:
    action: "Flush Cache"
    steps:
      - ssh deploy@myserver.example.com 'myapp/flush_cache.sh'
    allow_concurrency: true

Tasks like deploys can prompt for user defined environment variables:

tasks:
  restart:
    action: "Restart Application"
    description: "Sometimes needed if you want the application to restart but don't want to ship any new code."
    steps:
      - ssh deploy@myserver.example.com 'touch myapp/restart.txt'
    variables:
      -
        name: FORCE
        title: Restart server without waiting for in-flight requests to complete (Dangerous).
        default: 0

You can also make these variables appear in the task title:

tasks:
  failover:
    action: "Failover a pod"
    title: "Failover Pod %{POD_ID}"
    steps:
      - script/failover $POD_ID
    variables:
      - name: POD_ID

Review process

You can display review elements, such as monitoring data or a pre-deployment checklist, on the deployment page in Shipit:

review.checklist contains a pre-deploy checklist that appears on the deployment page in Shipit, with each item in the checklist as a separate string in the array. It can contain strong and a HTML tags. Users cannot deploy from Shipit until they have checked each item in the checklist.

For example:

review:
  checklist:
    - >
      Do you know if it is safe to revert the code being shipped? What happens if we need to undo this deploy?
    - Has the Docs team been notified of any major changes to the app?
    - Is the app stable right now?

review.monitoring contains a list of inclusions that appear on the deployment page in Shipit. Inclusions can either be images or iframes.

For example:

review:
  monitoring:
    - image: https://example.com/monitoring.png
    - iframe: https://example.com/monitoring.html

review.checks contains a list of commands that will be executed during the pre-deploy review step. Their output appears on the deployment page in Shipit, and if continuous delivery is enabled, deploys will only be triggered if those commands are successful.

For example:

review:
  checks:
    - bundle exec rake db:migrate:status

Shell commands timeout

All the shell commands can take an optional timeout parameter to limit their duration:

deploy:
  override:
    - ./script/deploy:
        timeout: 30
  post:
    - ./script/notify_deploy_end: { timeout: 15 }
review:
  checks:
    - bundle exec rake db:migrate:status:
        timeout: 60

See also commands_inactivity_timeout in secrets.yml for a global timeout setting.


Script parameters

Your deploy scripts have access to the following environment variables:

  • SHIPIT: Set to 1 to allow your script to know it's executed by Shipit
  • SHIPIT_LINK: URL to the task output, useful to broadcast it in an IRC channel
  • SHIPIT_USER: Full name of the user that triggered the deploy/task
  • EMAIL: Email of the user that triggered the deploy/task (if available)
  • ENVIRONMENT: The stack environment (e.g production / staging)
  • BRANCH: The stack branch (e.g master)
  • LAST_DEPLOYED_SHA: The git SHA of the last deployed commit
  • DIFF_LINK: URL to the diff on GitHub.
  • TASK_ID: ID of the task that is running
  • All the content of the secrets.yml env key
  • All the content of the shipit.yml machine.environment key

These variables are accessible only during deploys and rollback:

  • REVISION: the git SHA of the revision that must be deployed in production
  • SHA: alias for REVISION

Configuring providers

Heroku

To use Heroku integration (lib/snippets/push-to-heroku), make sure that the environment has Heroku CLI available.

Kubernetes

For Kubernetes, you have to provision Shipit environment with the following tools:

  • kubectl
  • kubernetes-deploy gem