A Build-Farm for Docker containers
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
docs
images
legacy
master
worker
.gitignore
README.md

README.md

chorus

http://chorus.niallcode.com

Chorus is a project for the Docker Community Hackthon. It has it's roots in a container scheduler called Orchestra.

Primer

Chorus is a clustered build farm for Docker containers. It's designed to manage building and running containers across a fleet of docker capable host machines.

You'll find Chorus useful if you have a number of machines using docker together, and you find yourself frequently building and re-building containers. (Which, can involve a lot of tweaking to get your builds just right...)

Conceptually, Chorus is very much like a private version of Dockerhub, with scheduler type functionality that let's you start the containers your build. This enables you to test the containers you build as part of your workflow.

There are two primary types of hosts:

  1. A Master Node which runs the chorus webservice and allows you to manage the fleet as a whole unit.
  2. One or more Worker Nodes which are simply machines functioning as dockerized hosts (meaning they can run docker containers.)

A large part of this project was the exploration of the dichotomy between using API's or Messaging Queues to handle inter-node communication inside the cluster. You can read about my findings here.

Overview

Chorus is a distributed build system that manages a group of machines that will build and run docker containers. It would work well on a fleet of machines in a Cloud Based environment operating on a common closed network (VPC in AWS speak). It will run on any LAN environment, and could conceivably run over the open internet in the future.

Overview

Components

Chorus uses a group of established technologies to provide the functionality required to manage a fleet of dockerized host machines:

  1. Python's Flask Framework for the Master-Node webservice
  2. Celery facilitates sending build jobs to Worker-Nodes, and for managing the main build queue.
  3. RabbitMQ for reliable and resiliant Inter-Node communication.
  4. PostgreSQL for managing persistant state information

Quick Start Guide

To deploy Chorus, you'll need machines with docker installed. Please consult the docker website for specifics, but in most cases it's as simple as: curl https://get.docker.com/ | sh

You'll probably want to use the bootstrap scripts to get up and running with a worker, so it's probably easiest to install git on your master, and worker nodes.

Once you've cloned the repo, you'll have to create config.json files, that contain the passwords needed. Here are two examples:

master config.json (should be placed in the chorus/master/ folder)

{
  "CHORUS_HOST": "THIS IS THE IP WORKERS WILL USE TO CONNECT TO CHORUS",
  "CHORUS_RMQ_MASTER": "THIS IS THE PASSWORD FOR THE MASTER QUEUE",
  "CHORUS_RMQ_WORKER": "THIS IS THE PASSWORD FOR THE WORKER QUEUE",
  "CHORUS_POSTGRES_PASSWORD": "THIS IS THE PASSWORD THAT WILL BE USED FOR POSTGRESQL"
}

worker config.json (should be placed in the chorus/worker/ folder)

{
  "CHORUS_HOST": "THIS IS THE IP WORKERS WILL USE TO CONNECT TO CHORUS",
  "CHORUS_RMQ_MASTER": "THIS IS THE PASSWORD FOR THE MASTER QUEUE",
  "CHORUS_RMQ_WORKER": "THIS IS THE PASSWORD FOR THE WORKER QUEUE",
  "CHORUS_POSTGRES_PASSWORD": "THIS IS THE PASSWORD THAT WILL BE USED FOR POSTGRESQL"
  "DOCKER_GID": "THIS IS THE NUMBER OF THE DOCKER GROUP GID, CAN BE FOUND BY CATTING /etc/groups"
}

Once you have created either a worker, or master config file you can launch either by running: $ ./bootstrap.sh (This must be done from the chorus/master or chorus/worker folders.)

If there are any problems, use the docker logs command in conjunction with docker ps -a to find your containers.