Breakdown of CloudFormation Template

This page describes the structure of CDeep3M's AWS CloudFormation template. This template provides a way to spin up AWS infrastructure via a web interface provided by AWS. For CDeep3M we are spinning up a GPU EC2 instance instance.

CDeep3M CloudFormation template is a file located here in the source tree: aws/basic_cloudformation.json and is in json format.

Click here for more information on CloudFormation templates

Overview

An empty CloudFormation template would look something like this:

{
 "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "<Put human readable description here>",
    "Parameters": { 
    },
    "Mappings": {
    },
    "Resources": {
    },
    "Outputs": {
    },
    "Metadata": {
    }
}

The rest of this document will break down version 1.6.2 of CDeep3M CloudFormation template file based on those main sections in the example above.

"Parameters": {}

In this section go the parameters that will be displayed to the user in the CloudFormation template webpage. Each parameter has a name, followed by Description, Type, and then optionally other parameters such as Default, ConstraintDescription, AlllowedPattern which lets the developer limit the types of inputs.

Actual parameters section from CDeep3M Click here for example of what user would see in web browser:

        "KeyName": {
            "Description": "Name of an existing EC2 KeyPair to enable SSH access to the instance",
            "Type": "AWS::EC2::KeyPair::KeyName",
            "ConstraintDescription": "must be the name of an existing EC2 KeyPair."
        },
        "DatasetURL": {
            "Description": "Web link that will be automatically downloaded by EC2 instance",
            "Type": "String",
            "Default": "",
            "AllowedPattern": "( *(\\b(https?|ftp)://)[-A-Za-z0-9+&@#/%?=~_|!:,.;]+)|( *)",
            "ConstraintDescription": "must be empty or set to a HTTP URL to a file for download"
        },
        "GPUInstanceType": {
            "Description": "GPU EC2 instance type",
            "Type": "String",
            "Default": "p3.2xlarge",
            "AllowedValues": [
                "p2.xlarge",
                "p3.2xlarge",
                "p3.8xlarge",
                "p3.16xlarge"
            ],
            "ConstraintDescription": "must be a valid EC2 GPU instance type."
        },
        "GPUDiskSize": {
            "Description": "Size of GPU EC2 instance disk in gigabytes",
            "Type": "Number",
            "Default": "100",
            "MinValue": "50",
            "ConstraintDescription": "must be a whole number 50 or larger"
        },
        "SSHLocation": {
            "Description": "The IP address range that can be used to SSH to the EC2 instances",
            "Type": "String",
            "MinLength": "9",
            "MaxLength": "18",
            "Default": "0.0.0.0/0",
            "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
            "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x."
        }

"Mappings": {}

To spin up an EC2 instance an Amazon AMI Image must be used. This section tells CloudFormation which AMI Image image to use. For CDeep3M section provides a mapping of which AMI Image to use for what region since an AMI Image can only be used in the region it is hosted in.

Instructions on how the mapping seen below can be found here

Actual parameters section from CDeep3M:

"Mappings" : {
       "RegionMap" : {
           "ap-south-1"    : { "AMI" : "ami-6a024a05"},
           "eu-west-1"    : { "AMI" : "ami-2e269e57"},
           "ap-northeast-2"    : { "AMI" : "ami-3097315e"},
           "ap-northeast-1"    : { "AMI" : "ami-fff07199"},
           "ap-southeast-1"    : { "AMI" : "ami-0746217b"},
           "ap-southeast-2"    : { "AMI" : "ami-fbfe0999"},
           "eu-central-1"    : { "AMI" : "ami-540f863b"},
           "us-east-1"    : { "AMI" : "ami-c4701dbe"},
           "us-east-2"    : { "AMI" : "ami-67dbf202"},
           "us-west-2"    : { "AMI" : "ami-bcce6ac4"}
       }       
    },

"Resources": {}

This is the most complicated part of the CDeep3M CloudFormation template and is where the EC2 instance is defined and the installation of CDeep3M code is done and to make it easier to parse this page I made a separate page for just this section.

Click here for Breakdown of Resources section of CloudFormation Template

NOTE: The file contains @@VERSION@@ tags which are updated by the dist target in the CDeep3M makefile

"Outputs": {}

This section contains the variables (and their values) that will be displayed to the user via the web browser page. The variables have a name (in example below InstanceId, AZ, PublicDNS..) and within a Description and a Value. The "REF": "GPUInstance" below is a CloudFormation function to get value of a variable and so is "Fn::GetAtt"

Actual parameters section from CDeep3M Click here for example of what user would see in web browser (See screenshot on Step 9):

"Outputs": {
        "InstanceId": {
            "Description": "InstanceId of the newly created EC2 instance",
            "Value": {
                "Ref": "GPUInstance"
            }
        },
        "AZ": {
            "Description": "Availability Zone of the newly created EC2 instance",
            "Value": {
                "Fn::GetAtt": [
                    "GPUInstance",
                    "AvailabilityZone"
                ]
            }
        },
        "PublicDNS": {
            "Description": "Public DNSName of the newly created EC2 instance",
            "Value": {
                "Fn::GetAtt": [
                    "GPUInstance",
                    "PublicDnsName"
                ]
            }
        },
        "PublicIP": {
            "Description": "Public IP address of the newly created EC2 instance",
            "Value": {
                "Fn::GetAtt": [
                    "GPUInstance",
                    "PublicIp"
                ]
            }
        }
    }

"Metadata": {}

I didn't touch this section and I believe its for the CloudFormation Designer.

Actual parameters section from CDeep3M:

    "Metadata": {
        "AWS::CloudFormation::Designer": {
            "85d8cb29-8b24-41c8-bca9-498f219d2b6e": {
                "size": {
                    "width": 60,
                    "height": 60
                },
                "position": {
                    "x": 60,
                    "y": 90
                },
                "z": 1,
                "embeds": []
            },
            "4ad8b7df-e9a7-4256-9ab2-9f548d4bb695": {
                "size": {
                    "width": 60,
                    "height": 60
                },
                "position": {
                    "x": 180,
                    "y": 90
                },
                "z": 1,
                "embeds": [],
                "isassociatedwith": [
                    "85d8cb29-8b24-41c8-bca9-498f219d2b6e"
                ]
            }
        }
    }