Skip to content
Auto-scaling Concourse CI v2.2.1 on AWS with Terraform (since 2016.04.14)
Go HCL Shell Makefile
Branch: master
Clone or download
Latest commit c0f385a Aug 24, 2017
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
autoscaling Added necessary prefixes to resources in autoscaling/hooks/enabled May 26, 2016
ci
cmd Update URL for myexternalIP to use https Jun 30, 2017
concourse format code Jan 24, 2017
postgres First commit Apr 14, 2016
scripts Bump Concourse CI to v1.1 released today Apr 15, 2016
.gitignore
00_debug.sh First commit Apr 14, 2016
00_install_concourse.sh.tpl First commit Apr 14, 2016
01_start_concourse_web.sh.tpl
02_start_concourse_worker.sh.tpl fixes #15. bump concourse to 2.5.1 Dec 24, 2016
LICENSE `concourse-aws` command to ease Concourse cluster configuration in in… May 19, 2016
Makefile version 0.0.3 Aug 10, 2016
README.md fix download link in README Aug 24, 2017
build-amis.sh
build-concourse-ami.sh Concourse v1.3.0-rc.83 Jun 4, 2016
build-docker-ami.sh fix CVE-2017-1000117 Aug 25, 2017
build-ubuntu-ami.sh Concourse v1.3.0-rc.83 Jun 4, 2016
concourse-baked.json iam roles cannot be used when packer runs Feb 14, 2017
docker-baked.json fix CVE-2017-1000117 Aug 25, 2017
glide.lock
glide.yaml
hello.yml
latest-ami-docker.sh Concourse v1.3.0-rc.83 Jun 4, 2016
latest-ami-ubuntu.sh Concourse v1.3.0-rc.83 Jun 4, 2016
main.go `concourse-aws` command to ease Concourse cluster configuration in in… May 19, 2016
main.tf
my-latest-ami.sh First commit Apr 14, 2016
outputs.tf support https on elb Jul 14, 2016
packer.json fix CVE-2017-1000117 Aug 25, 2017
recreate.sh ASG level blue-green deployment of web/worker Apr 15, 2016
terraform.sh
variables.tf
version v0.0.5 Jan 24, 2017

README.md

Auto-scaling Concourse CI on AWS with Terraform

http://www.slideshare.net/mumoshu/autoscaled-concourse-ci-on-aws-wo-bosh

Recommended Usage: Using concourse-aws binary

  1. Install packer and terraform(< 0.7.0)
  1. Create 1 VPC and 2 subnets in it

  2. Clone this repository

git clone https://github.com/mumoshu/concourse-aws
  1. Download latest concourse-aws binary from Github releases and place the binary in your concourse-aws directory.

  2. Run concourse-aws up

cd concourse-aws

./build-amis.sh

./concourse-aws up

And then, concourse-aws will prompt you to provide required parameters(region, availability zone, subnet id, cidr, and vice versa)

Upgrading Concourse workers with latest binaries

$ git pull --rebase origin master
$ ./build-concourse-ami.sh
$ vi cluster.yml # and update `ami_id` with the one produced by `build-concourse-ami.sh`
$ ./concourse-aws up

Syncing configurations and states with S3

Users would sometime like to save and restore configurations and states on AWS resources, which is actually managed by terraform, with external data storage. concourse-aws supports to save/restore states files with S3. You can these operation with save/restore command like below:

# Save
# this will save configurations and states on AWS resources to S3 bucket.
./concourse-aws save --bucket <bucket_name> --bucket-region <region_of_bucket>
# restore
# this will pull configurations and states on AWS resources to S3 bucket.
./concourse-aws restore --bucket <bucket_name> --bucket-region <region_of_bucket>

Note: Saved/restored files

  • cluster.yml: configuration file which can be generated by concourse-aws interactively.
  • SSH keys used for communicating between concourse servers. These key files can be also automatically generated by concourse-aws interactively.
  • host_key,host_key.pub
  • worker_key,worker_key.pub
  • session_signing_key,session_signing_key.pub
  • authorized_worker_keys
  • terraform.tfstate
    • states of AWS resources managed by terraform

Advanced Usage: Using shell scripts and terraform directly

  1. Install packer and terraform

  2. Create 1 VPC and 2 subnets in it

  3. Set up required environment variables required by the wrapper script for terraform

    $ cat >> .envrc <<<'
    export AWS_ACCESS_KEY_ID=<YOUR ACCESS KEY>
    export AWS_SECRET_ACCESS_KEY=<YOUR SECRET ACCESS KEY>
    export CONCOURSE_IN_ACCESS_ALLOWED_CIDRS="<YOUR_PUBLIC_IP1>/32,<YOUR_PUBLIC_IP2>/32"
    export CONCOURSE_SUBNET_ID=<YOUR_SUBNET1_ID>,<YOUR_SUBNET2_ID>
    export CONCOURSE_DB_SUBNET_IDS=<YOUR_SUBNET1_ID>,<YOUR_SUBNET2_ID>
    '
    

    Install direnv and allow it to read .envrc created in the previous step.

    $ direnv allow
    
  4. The same for optional ones

    $ export CONCOURSE_WORKER_INSTANCE_PROFILE=<YOUR INSTANCE PROFILE NAME>
    
  5. Edit terraform variables and Run the following commands to build required AMIs and to provision a Concourse CI cluster

    $ ./build-amis.sh
    $ vi ./variables.tf
    $ ./terraform.sh get
    $ ./terraform.sh plan
    $ ./terraform.sh apply
    
  6. Open your browser and confirm that the Concourse CI is running on AWS:

    # This will extract the public hostname for your load balancer from terraform output and open your default browser
    $ open http://$(terraform output | ruby -e 'puts STDIN.first.split(" = ").last')
    
  7. Follow the Concourse CI tutorial and experiment as you like:

    $ export CONCOURSE_URL=http://$(terraform output | ruby -e 'puts STDIN.first.split(" = ").last')
    $ fly -t test login -c $CONCOURSE_URL
    $ fly -t test set-pipeline -p hello-world -c hello.yml
    $ fly -t test unpause-pipeline -p hello-world
    

    See http://concourse.ci/hello-world.html for more information and the hello.yml referenced in the above example.

  8. Modify autoscaling groups' desired capacity to scale out/in webs or workers.

Why did you actually created this?

BOSH looks very promising to me according to what problems it solves. However I was too lazy to learn it for now mainly because:

  • I'm not going to use IaaS other than AWS for the time being
  • learning it to JUST try Concourse CI might be too much in the short term though

You may also find those projects useful

Contributing

Making changes

The concourse-aws binary needs to be built for every architecture and pushed to GitHub Releases manually whenever the concourse-aws binary has code changes. Every significant change to the functionality should result in a bump of the version number in the version file.

You can’t perform that action at this time.