UnixDaemon Experiments Terraform Repo

Introduction

While it's possible to experiment and learn parts of Terraform in isolation sometimes it's handy to have a larger, more complete, environment to run your tests in. For me unixdaemon_terraform_experiments this is that repo. It will contain a number of different terraform based projects that can be consistently deployed together. You can see some of my thinking behind this in the Naive first steps with Terraform post.

Terraform is a very powerful, but quite young, piece of software so I'm making this repo open to encourage sharing and invite feedback on better way to do things. There is no guarantee that anything in this repo is the best or most current way to do anything.

Bootstrap

The bootstrap phase requires you to have AWS account credentials. For this repo it's recommended that you store them in .aws/credentials under distinct profile names and leave [default] empty.

We'll do the initial terraform configuration out of bounds to avoid making bootstrapping difficult. First we create the S3 bucket, which must have a globally unique name, used to store the terraform state files. Then we enable bucket versioning in case of anything going hideously wrong.

The AWS_REGION and DEPLOY_ENV variables will help us when we later need to have AWS resources in multiple regions or if you decide to have separate test, staging and production environments for example.

export AWS_PROFILE=test-admin
export AWS_REGION=eu-west-1
export DEPLOY_ENV=test

export TERRAFORM_BUCKET="net.dean-wilson-terraform-state-${AWS_REGION}-${DEPLOY_ENV}"

$ aws --region $AWS_REGION s3 mb "s3://${TERRAFORM_BUCKET}"
make_bucket: s3://net.dean-wilson-terraform-state-eu-west-1-test/

$ aws --region $AWS_REGION       \
    s3api put-bucket-versioning  \
    --bucket ${TERRAFORM_BUCKET} \
    --versioning-configuration Status=Enabled

You will also need to make a change to the projects Rakefile and tell it your BUCKET_NAME and BUCKET_REGION`. These are (currently, and awkwardly) set as constants at the top of the file and should match the values you exported above.

You should now install Terraform. This can be done by downloading the file from the Terraform website, or possibly installing it using your package manager.

Once this is done we'll enable our rake terraform wrapper by installing its dependencies.

$ bundle install

You can then see the possible rake tasks with

$ bundle exec rake -T
...
rake plan                  # Show the terraform plan
...

Setting up an environment

Before we add our first Terraform project we'll configure an environment. I've decided to structure this repo and code to have three environments, test, staging and production. Each of those will be implemented as a distinct Amazon AWS Account and will have their own S3 distinct bucket for state. If you want to have your own environment names then you'll need to change ALLOWED_ENVIRONMENTS in the Rakefile.

We then create our environment specific variable file.

mkdir variables

echo 'environment = "test"' > variables/test.tfvars

Running an initial terraform project

Now we're past all the basic configuration we'll create a very simple Terraform project and apply it to confirm everything is working. For our initial project we'll create a security group and then delete it to show the entire end to end process.

Our initial step is to create a directory under projects to hold our new resources. Once this is done we'll add a single security group resource.

mkdir -p projects/simple-sg/resources/

cat > projects/simple-sg/resources/security-group.tf <<EOF
resource "aws_security_group" "test_sg" {
    name = "test-labs-sg"
    description = "A test-labs-sg example resource"
}
EOF

Now everything is configured and we have a simple test case we'll run Terraform and see check if everything works.

$ PROJECT_NAME=simple-sg bundle exec rake plan

Remote state configured and pulled.
...
+ aws_security_group.test_sg
    description: "" => "A test-labs-sg resource"
    name:        "" => "test-labs-sg"
...
Plan: 1 to add, 0 to change, 0 to destroy.

Everything is looking good so far. Terraform has now shown us what it will do when we apply it to our infrastructure. Which we'll do now.

# notice that the rake task changes from plan to apply
$ PROJECT_NAME=simple-sg bundle exec rake apply

aws_security_group.test_sg: Creating...
  description: "" => "A test-labs-sg resource
...
aws_security_group.test_sg: Creation complete
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

You can now check for the security group using either the AWS console or the aws command line tool. If you now check the S3 bucket we configured and created earlier to store our Terraform statefiles you'll also see an object named terraform-simple-sg.tfstate. This is where Terraform stores its remote state.

If you re-run the terraform apply nothing will will change as Terraform has no more work to do.

$ PROJECT_NAME=simple-sg bundle exec rake apply

aws_security_group.test_sg: Refreshing state... (ID: sg-000000)
Apply complete! Resources: 0 added, 0 changed, 0 destroyed

We'll now finish our testing and clean up after ourselves by having Terraform destroy the resource we created.

$ PROJECT_NAME=simple-sg bundle exec rake destroy

Do you really want to destroy?
  Terraform will delete all your managed infrastructure.

  Enter a value: yes

aws_security_group.test_sg: Refreshing state... (ID: sg-000000)
aws_security_group.test_sg: Destroying...
aws_security_group.test_sg: Destruction complete

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Adding Tests

While Terraform is remarkably good at its job there are going to be some occasions when you want to test what you wanted actually happened. In the unixdaemon_terraform_experiments repository I'm handling this with awspec and a little custom rspec directory modification.

First we pull in the awspec gem.

bundle install

We also need to add the necessary scaffolding files:

echo "gem 'awspec',  '~> 0.37'" >> Gemfile

mkdir spec

echo "require 'awspec'" >> spec/spec_helper.rb

Now we'll add a test to our simple-sg project to confirm that the security group was created.

mkdir projects/simple-sg/spec

$ cat > projects/simple-sg/spec/security_group_spec.rb <<EOF
require 'spec_helper'

describe security_group('test-labs-sg') do
  it { should exist }
end
EOF

Note that the tests live beside the terraform project resources, not in a combined spec directory. This allows us to run only the tests related to the project we're currently working on.

We then use rake spec to run tests against our chosen project.

PROJECT_NAME=simple-sg bundle exec rake spec

As we tidied up after ourselves previously this spec run will fail.

PROJECT_NAME=simple-sg bundle exec rake spec

security_group 'test-labs-sg'
  should exist (FAILED - 1)

Finished in 0.03664 seconds (files took 1.67 seconds to load)
1 example, 1 failure

We'll now recreate the security group and then verify that it exists with the name we gave it.

$ PROJECT_NAME=simple-sg bundle exec rake apply
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.


$ PROJECT_NAME=simple-sg bundle exec rake spec

security_group 'test-labs-sg'
  should exist

Finished in 0.00153 seconds (files took 1.36 seconds to load)
1 example, 0 failures

Don't forget to destroy the security group when you're done testing.

Something to consider is that you don't want to duplicate all your terraform work and retest your resource declarations. Instead you should test more dynamic aspects of your configuration. Verifying a templated policy contains the expected strings or that all policies have been attached to a group are much better things to test than just the existence of a resource.

I think awspec is a wonderful little tool and I can see it being useful both when migrating from Ansible to Terraform and to later verify my newer projects.

Future Plans

In general most of the things under projects will start out as tf files full of resources and will be extracted as modules when they're generic enough to be useful on their own. I'm sure there is third party code I could use to build most of this but as this repo is mostly for learning it'd be a little counter productive to import too much functionality.

I'll be expanding the repo with working examples as I hit new use cases and hopefully having a larger, related, chunk of terraform code will be useful to people new to Terraform.

Since you've made it all the way here another link to the repo might be in order: unixdaemon_terraform_experiments