AWS: CloudFormation

Hints

• The template is the primary CloudFormation artifact. It defines the system that will be provisioned. It is JSON, with a mechanism for adding comments (funny, I came up a similar configuration scheme for udp_forwarder).
• There are several ways that you can initialize a template. You can start with an existing template, and you can have the CloudFormer tool create a template from an existing AWS instance. I first tried CloudFormer, but it was very complicated and wanted to create a a bunch of specifications that didn't mean anything to me. Finally, I just started with a Rails sample template, and edited this. It was quite straightforward.
• Disable 'Rollback' (in Advanced options) in order to preserve the instance if the provisioning fails. Then you can log in and check the log files to find out what went wrong.
• It's easy to miss selection of the "KeyName" on the parameters page. You need to provide this if you want to be able to ssh into the running system.
• tail -f /var/log/cfn-init.log seems to be the best way to track the provisioning.
• You will see the CloudFormation stack instances listed on the CloudFormation service page, as well as in the EC2 dashboard.
• If you stop or restart the instance, a new IP will be assigned to it.

What Happens When Creating a Stack

You mash the Create Stack button. This is roughly what happens:

1. CloudFormation parses your template. The items in the Parameters section are extracted.
2. You are presented with configuration pages, letting you set or override defaults for the parameters.
3. Provisioning for each item in the Resources is begun.
4. When the resources are provisioned, items defined on the Output section are displayed in the output tab.

The process of provisioning the CHORDS EC2 instance (aka WebServer) is roughy as follows:

1. WebServer::Properties::ImageId specifies an AMI that will be instantiated.
2. WebServer::Properties::UserData contains a script that will be executed after the instance is running. For CHORDS, this script then runs cfn-init.
3. cfn-init follows the recipe provided in *WebServer::MetaData::AWS::CloudFormation::Init to install all of CHORDS infrastructure. The AWS::CloudFormation::Init reference describes the format and functions of the JSON specification.

The Template

Notes

• The key section is the resource MetaData, which is used by cfn-init. The most useful reference is the AWS::CloudFormation::Init reference.
• Being JSON, it's really easy to create a malformed template. There are tools, such as the AWS Eclipse add-on, which can verify well formed JSON.
• However, it seems that the only way to debug a template is to repeatedly create a new stack, ssh into the system, and examine the files in /var/log/. (You did remember to select the AWS KeyPair, and to disable rollback?).
• Configsets have a number of sections, which define different families of actions. The most important fact is the order in which these are executed, which the documentations tells us:

The cfn-init helper script processes these configuration sections in the following order: packages, groups, users, sources, files, commands, and then services. If you require a different order, separate your sections into different config keys, and then use a configset that specifies the order in which the config keys should be processed.

Structure

The AWS CloudFormation Reference is the ultimate source of information. However, it doesn't give a very good "big picture" overview of how everything fits together. Here are some concepts that I've gleaned.

The template anatomy is a good place to start.

The only required section is Resources, but our template contains the following top-level sections:

• Description: A concise summary of what the stack provides.
• Parameters: User settable parameters that will appear during the CloudFormation stack creation process.
• Mappings: Simply a set of hashes that other parts of the template can reference.
• Resources: This is the main section, with multiple entries. Each resource defines an AWS resource that will be provisioned or referenced. Thus, you might have resources defined for an EC2 instance, an ElasticIP value, an S3 bucket, etc. You can provision multiple EC2 Instances if that is your stack architecture design.
• Outputs: Defines results from the stack creation process that will appear in the Outputs tab of the CloudFormation console.

The EC2 Instance Resource

The CHORDS template specifies one machine image resource, which is named WebServer. Within WebServer are the following sections:

• Type: Specifies an EC2 Instance.
• Properties: Information on how to build the EC2 instance. The name of a canned AWS AMI is determined, security groups are assigned, and so on. The UserData section specifies a script which will be run once when the instance is finally running. This script initiates installation of the application infrastructure.
• MetaData: This specifies a JSON structure that will be available to the UserData script, once the EC2 instance is running. For CHORDS, the UserData calls cfn-int, which uses this information. This is where the bulk of the application bootstrapping takes place.
• CreationPolicy: Specifies a time limit allowed for stack creation.

Eclipse

• You can easily survive without using Eclipse, just using the online AWS management tools.
• You can install the AWS Tools For Eclipse, which provides an Eclipse interface to AWS.
• However, access credentials (Access Key ID and a Secret Key) are required for this remote access. These are part of the AWS IAM system. You can go to that page, select the user, and see a list of Access Key ID. The Secret Key is only available once, when you create an Access Key. Save both in a secure location.

Standalone AWS CLI

The AWS command line program can be used to interact with stacks. It must be installed, and then configured with the secret key id and the secret key. These are created on the security tab for a user. If you loose the key, you can delete an existing key and create a new id/key pair. Note: these credentials are not the same as the RSA ssh key found in the .pem file.

Install the AWS CLI.
Configure AWS CLI.

However you really can't seem to do much more with these than what you can get through the cloudformation web interface.

There are instructions in the AWS documentation for having the cfn logs saved to S3, so that they can be viewed if a cloudformation stack creation fails. The whole procedure is arcane and complicated. I never tried to get it to work.