An example of how to provision a CoreOS cluster on AWS using Terraform and ansible-vault.
HCL Shell
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
ca
cluster
example-api
nginx-proxy
rds
registry
s3
scripts
vault
vpc
.gitignore
LICENSE
README.md
ansible.cfg
aws.tf
main.tf
outputs.tf
requirements.txt
variables.tf
vault_password.gpg

README.md

CoreOS Cluster

An example of how to provision a CoreOS cluster on AWS using Terraform. This example sets up a VPC, private and public networks, NAT server, an RDS database, a CoreOS cluster and a private Docker registry and properly configures tight security groups.

The cluster is configured via cloud-config user data and runs etcd2.service and fleet.service. All peer and client traffic is encrypted using self signed certificates.

A private Docker registry is also created at registry.local and Docker nodes are properly configured to use it.

Index

Requirements

You need Terraform >= 0.7.0, e.g.:

brew install terraform

You need an SSH key. The private key needs to be chmod to 600.

You need the cloud provider credentials. These will be entered on the command line.

Vault

Setting Up GPG-encrypted Vault Support

You will need to have setup gpg-agent on your computer before you start.

brew install gpg
brew install gpg-agent

If you haven't already generated your PGP key (it's ok to accept the default options if you never done this before):

gpg --gen-key

Get your KEYID from your keyring:

gpg --list-secret-keys | grep sec

This will probably be pre-fixed with 2048R/ or 4096R/ and look something like 93B1CD02.

Send your public key to PGP key server:

gpg --keyserver pgp.mit.edu --send-keys KEYID

To import a public key (e.g. when a new engineer joins the team):

gpg --keyserver pgp.mit.edu --search-keys john@doe.com

Create ~/.bash_gpg:

envfile="${HOME}/.gnupg/gpg-agent.env"

if test -f "$envfile" && kill -0 $(grep GPG_AGENT_INFO "$envfile" | cut -d: -f 2) 2>/dev/null; then
  eval "$(cat "$envfile")"
else
  eval "$(gpg-agent --daemon --log-file=~/.gpg/gpg.log --write-env-file "$envfile")"
fi
export GPG_AGENT_INFO  # the env file does not contain the export statement

Add to ~/.bash_profile:

GPG_AGENT=$(which gpg-agent)
GPG_TTY=`tty`
export GPG_TTY

if [ -f ${GPG_AGENT} ]; then
  . ~/.bash_gpg
fi

Start a new shell or source the current environment:

source ~/.bash_profile

Encrypt / Decrypt Vault Password

Encrypt the vault password:

echo "the vault password" | gpg -e -r "your_email_address" > vault_password.gpg

Ansible Vault will decrypt the file based using PGP key from your keyring. See vault_password_file option in the ansible.cfg configuration file.

Create SSL Certificate

An SSL certificate can be purchased from Comodo.

Comodo will then provide the certificate itself in a ZIP file which consists of:

  • certificate (STAR_your_domain_com.crt)
  • intermediate and root Comodo certificates

AWS only supports PEM format of certificates so first, we needed to convert the private key and all certificates to the correct format:

openssl x509 -inform PEM -in your_domain_com.key > your_domain_com.pem
openssl x509 -inform PEM -in STAR_your_domain_com.crt > STAR_your_domain_com.pem
openssl x509 -inform PEM -in COMODORSADomainValidationSecureServerCA.crt > COMODORSADomainValidationSecureServerCA.pem
openssl x509 -inform PEM -in COMODORSAAddTrustCA.crt > COMODORSAAddTrustCA.pem
openssl x509 -inform PEM -in AddTrustExternalCARoot.crt > AddTrustExternalCARoot.pem

Secondly, generate a certificate chain from intermediate and root certificates one by one using cat command:

cat COMODORSADomainValidationSecureServerCA.pem COMODORSAAddTrustCA.pem AddTrustExternalCARoot.pem > your_domain_com_certificate_chain.pem

Generate Deployer Key Pair

You will need to generate a deployer key pair. The public key will be added to all EC2 instances:

ssh-keygen -t rsa -b 4096 -C "your@email"

Required Secure Variables

This repository is using ansible-vault to secure sensitive information. Secure variables for each environment are stored in a separate file in vault directory:

.
├── vault
│   ├── stage.yml
│   └── prod.yml
│
└── ...

If you already know the password you do not need to recreate the vault/<env-name-prefix>.yml file.

You can edit variables stored in the vault:

ansible-vault edit vault/<env-name-prefix>.yml

Required contents for vault/<env-name-prefix>.yml (if you don't know the password):

aws_region: "eu-west-1"
is_development: true
dns_zone_id: "Z1234"
dns_zone_name: "yourdomain.com"
api:
  database_type: "postgres"
  database_host: "database1.local"
  database_port: 5432
  database_user: "example_api"
  database_password: "test_password"
  database_name: "example_api"
  database_max_idle_conns: 5
  database_max_open_conns: 5
  scheme: "https"
  host: "localhost:8080"
  dns_prefix: "<environment>-api."
app:
  scheme: "https"
  host: "localhost:8000"
public_key: ssh-rsa ...
ssl_certificate_body: |
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----
ssl_certificate_chain: |
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----
ssl_certificate_private_key: |
  -----BEGIN RSA PRIVATE KEY-----
  ...
  -----END RSA PRIVATE KEY-----

AWS Provisioning

Environment Variables

The terraform provider for AWS will read the standard AWS credentials environment variables. You must have these variables exported:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_DEFAULT_REGION (eu-west-1)

You can get the credentials from the AWS console.

SSH Agent

Terraform will look for a deployment key in ~/.ssh directory when creating a NAT instance. Add the deployment key to the ssh-agent, e.g.:

ssh-add ~/.ssh/stage-deployer

Provisioning

Render an SSH configuration file, i.e.:

./scripts/render-ssh-config.sh <env-name-prefix> <domain-name>

Create virtual Python environment:

virtualenv .venv
source .venv/bin/activate
pip install -r requirements.txt

Variables

You will need to export couple of needed environment variables.

Most importantly, define an environment name, e.g.:

export TF_VAR_env=stage

Or:

export TF_VAR_env=prod

Export DNS variables:

export TF_VAR_dns_zone_id=$(./scripts/get-vault-variable.sh $TF_VAR_env dns_zone_id)
export TF_VAR_dns_zone_name=$(./scripts/get-vault-variable.sh $TF_VAR_env dns_zone_name)
export TF_VAR_api_dns_prefix=$(./scripts/get-vault-variable.sh $TF_VAR_env api.dns_prefix)

Export public key:

export TF_VAR_public_key=$(./scripts/get-vault-variable.sh $TF_VAR_env public_key)

Export SSL certificate:

export TF_VAR_ssl_certificate_body=$(./scripts/get-vault-variable.sh $TF_VAR_env ssl_certificate_body)
export TF_VAR_ssl_certificate_chain=$(./scripts/get-vault-variable.sh $TF_VAR_env ssl_certificate_chain)
export TF_VAR_ssl_certificate_private_key=$(./scripts/get-vault-variable.sh $TF_VAR_env ssl_certificate_private_key)

Export DB variables from the ansible-vault:

export TF_VAR_db_name=$(./scripts/get-vault-variable.sh $TF_VAR_env api.database_name)
export TF_VAR_db_user=$(./scripts/get-vault-variable.sh $TF_VAR_env api.database_user)
export TF_VAR_db_password=$(./scripts/get-vault-variable.sh $TF_VAR_env api.database_password)

Load JSON configuration for API service:

export TF_VAR_api_config=$(./scripts/get-api-config.sh $TF_VAR_env)

For test environments, it's useful to disable final DB snapshot:

export TF_VAR_rds_skip_final_snapshot=true

See variables.tf for the full list of variables you can set.

State Files

We need to support multiple environments (stage, prod etc) and share state files across the team. Therefor state files include environment suffix and their encrypted versions are stored in git.

First, decrypt a state file you want to use:

./scripts/decrypt-state-file.sh $TF_VAR_env

The above script would decrypt $TF_VAR_env.tfstate.gpg to $TF_VAR_env.tfstate.

After running Terraform, don't forget to update the encrypted state file:

./scripts/encrypt-state-file.sh $TF_VAR_env

Apply Execution Plan

First, check the Terraform execution plan:

terraform plan -state=$TF_VAR_env.tfstate

Now you can provision the environment:

terraform apply -state=$TF_VAR_env.tfstate

IMPORTANT: The option -var force_destroy=true will mark all the resources, including S3 buckets to be deleted when destroying the environment. This is fine in test environments, but dangerous in production.

So, you could provision a test environment like this:

terraform apply -state=$TF_VAR_env.tfstate -var force_destroy=true

Connecting To Instances

Now you can SSH to instances in a private subnet of the VPS via the NAT instance, e.g.:

ssh -F ssh.config <private_ip>

Debugging

Once you have connected to a CoreOS node, you can use systemctl to check status of services:

systemctl status api.service

To view logs specific to a service running as a Docker container, use journalctl:

journalctl -u api.service -n 100 -f

Recreate Database

NEVER do this against production environment.

Sometimes during development it is useful or needed to destroy and recreate the database. You can use taint command to mark the database instance for destruction. Next time you run the apply command the database will be destroyed and a new once created:

terraform taint -module=rds -state=$TF_VAR_env.tfstate aws_db_instance.rds

Managing Cluster

Once your cluster is app and running, there are plenty of ways to manage it.

Private Docker Registry

The cluster is configured to use a private Docker registry secured by a self signed certificate. It uses S3 as a storage backend.

To push images to the registry, ssh to the instance and do something like:

git clone https://github.com/RichardKnop/example-api.git
cd example-api
./build-release.sh v0.0.0 --no-dry-run -y

You can then pull the image from cluster nodes:

docker pull registry.local/example-api:latest

ETCD

Inspect cluster health:

etcdctl cluster-health

Fleet

List machines:

fleetctl list-machines

List units:

fleetctl list-units

Start a unit:

fleetctl start example-api.service

Resources