Skip to content
Auto-scaling Concourse CI v2.2.1 on AWS with Terraform (since 2016.04.14)
Go HCL Shell Makefile
Branch: master
Clone or download
Latest commit c0f385a Aug 24, 2017
Type Name Latest commit message Commit time
Failed to load latest commit information.
autoscaling Added necessary prefixes to resources in autoscaling/hooks/enabled May 26, 2016
cmd Update URL for myexternalIP to use https Jun 30, 2017
concourse format code Jan 24, 2017
postgres First commit Apr 14, 2016
scripts Bump Concourse CI to v1.1 released today Apr 15, 2016
.gitignore First commit Apr 14, 2016 First commit Apr 14, 2016 fixes #15. bump concourse to 2.5.1 Dec 24, 2016
LICENSE `concourse-aws` command to ease Concourse cluster configuration in in… May 19, 2016
Makefile version 0.0.3 Aug 10, 2016 fix download link in README Aug 24, 2017 Concourse v1.3.0-rc.83 Jun 4, 2016 fix CVE-2017-1000117 Aug 25, 2017 Concourse v1.3.0-rc.83 Jun 4, 2016
concourse-baked.json iam roles cannot be used when packer runs Feb 14, 2017
docker-baked.json fix CVE-2017-1000117 Aug 25, 2017
hello.yml Concourse v1.3.0-rc.83 Jun 4, 2016 Concourse v1.3.0-rc.83 Jun 4, 2016
main.go `concourse-aws` command to ease Concourse cluster configuration in in… May 19, 2016 First commit Apr 14, 2016 support https on elb Jul 14, 2016
packer.json fix CVE-2017-1000117 Aug 25, 2017 ASG level blue-green deployment of web/worker Apr 15, 2016
version v0.0.5 Jan 24, 2017

Auto-scaling Concourse CI on AWS with Terraform

Recommended Usage: Using concourse-aws binary

  1. Install packer and terraform(< 0.7.0)
  1. Create 1 VPC and 2 subnets in it

  2. Clone this repository

git clone
  1. Download latest concourse-aws binary from Github releases and place the binary in your concourse-aws directory.

  2. Run concourse-aws up

cd concourse-aws


./concourse-aws up

And then, concourse-aws will prompt you to provide required parameters(region, availability zone, subnet id, cidr, and vice versa)

Upgrading Concourse workers with latest binaries

$ git pull --rebase origin master
$ ./
$ vi cluster.yml # and update `ami_id` with the one produced by ``
$ ./concourse-aws up

Syncing configurations and states with S3

Users would sometime like to save and restore configurations and states on AWS resources, which is actually managed by terraform, with external data storage. concourse-aws supports to save/restore states files with S3. You can these operation with save/restore command like below:

# Save
# this will save configurations and states on AWS resources to S3 bucket.
./concourse-aws save --bucket <bucket_name> --bucket-region <region_of_bucket>
# restore
# this will pull configurations and states on AWS resources to S3 bucket.
./concourse-aws restore --bucket <bucket_name> --bucket-region <region_of_bucket>

Note: Saved/restored files

  • cluster.yml: configuration file which can be generated by concourse-aws interactively.
  • SSH keys used for communicating between concourse servers. These key files can be also automatically generated by concourse-aws interactively.
  • host_key,
  • worker_key,
  • session_signing_key,
  • authorized_worker_keys
  • terraform.tfstate
    • states of AWS resources managed by terraform

Advanced Usage: Using shell scripts and terraform directly

  1. Install packer and terraform

  2. Create 1 VPC and 2 subnets in it

  3. Set up required environment variables required by the wrapper script for terraform

    $ cat >> .envrc <<<'

    Install direnv and allow it to read .envrc created in the previous step.

    $ direnv allow
  4. The same for optional ones

  5. Edit terraform variables and Run the following commands to build required AMIs and to provision a Concourse CI cluster

    $ ./
    $ vi ./
    $ ./ get
    $ ./ plan
    $ ./ apply
  6. Open your browser and confirm that the Concourse CI is running on AWS:

    # This will extract the public hostname for your load balancer from terraform output and open your default browser
    $ open http://$(terraform output | ruby -e 'puts STDIN.first.split(" = ").last')
  7. Follow the Concourse CI tutorial and experiment as you like:

    $ export CONCOURSE_URL=http://$(terraform output | ruby -e 'puts STDIN.first.split(" = ").last')
    $ fly -t test login -c $CONCOURSE_URL
    $ fly -t test set-pipeline -p hello-world -c hello.yml
    $ fly -t test unpause-pipeline -p hello-world

    See for more information and the hello.yml referenced in the above example.

  8. Modify autoscaling groups' desired capacity to scale out/in webs or workers.

Why did you actually created this?

BOSH looks very promising to me according to what problems it solves. However I was too lazy to learn it for now mainly because:

  • I'm not going to use IaaS other than AWS for the time being
  • learning it to JUST try Concourse CI might be too much in the short term though

You may also find those projects useful


Making changes

The concourse-aws binary needs to be built for every architecture and pushed to GitHub Releases manually whenever the concourse-aws binary has code changes. Every significant change to the functionality should result in a bump of the version number in the version file.

You can’t perform that action at this time.