Helps you deploy AWS infrastructure with Cloudformation.
This project is originally based on the cumulus project. See CUMULUS_LICENSE for license information.
Install the AWS CLI:
pip install awscli
Configure the AWS CLI:
aws configure
humilis
will use whatever credentials you introduced when configuring your
AWS CLI installation.
You can now install the latest "stable" version of humilis
:
pip install humilis
or the development version if you prefer that:
pip install git+https://github.com/humilis/humilis
After installation you need to configure humilis. To configure globally for your system:
humilis configure
The command above will store and read the configuration options from
~/.humilis.ini
. You can also store the configuration in a .humilis.ini
file
stored in your current working directory by using:
humilis configure --local
humilis
will always read the configuration first from a .humilis.ini
file
under your current work directory. If it is not found then it will read it from
your system global config file ~/.humilis
.
Assuming you have virtualenv installed:
make develop
. .env/bin/activate
At the moment, most tests are integration tests with the AWS SDK. This means that you will need to set up your system to access AWS resources if you want to run the test suite.
py.test tests
Define your infrastructure environment following the examples in the examples directory. Then to create the environment:
humilis create examples/humilis-firehose.yaml
To update the environment after it has been deployed:
humilis update examples/humilis-firehose.yaml
And to delete it:
humilis delete examples/humilis-firehose.yaml
A humilis
environment is just a collection of cloudformation stacks that
are required for an application. Instead of having a monolytic CF template for
your complete application, humilis
allows you to define infrastructure
layers that are combined into an environment. Each humilis
layer
translates exactly into one CF template (therefore into one CF stack after
the layer is deployed).
Breaking a complex infrastructure environment into smaller layers has at least two obvious advantages:
-
Easier to maintain. It's easier to maintain a simple layer that contains just a bunch of CF resources than serve a well-defined purpose.
-
Easier to reuse. You should strive to define your infrastructure layers in such a way that you can reuse them across various environments. For instance, many projects may require a base layer that defines a VPC, a few subnets, a gateway and some routing tables, and maybe a (managed) NAT. You can define a humilis layer with those resources and have a set of layer parameters (e.g. the VPC CIDR) that will allow you to easily reuse it across environments.
An environment definition file is a yaml document that specifies the
list of layers that form your enviroment. The file should be named as your
environment. That is, for environment my-app-environment
the environment
description file should be called my-app-environment.yaml
. The contents of
the environment definition should be organized as follows:
---
my-app-environment:
description:
A description of what this environment is for
layers:
# The layers that you environment requires. They will be deployed in the
# same order as you list them. Note that you can also pass parameters
# to a layer (more on that later).
- {layer: name_of_first_layer, layer_param: layer_value}
- {layer: name_of_second_layer}
- {layer: name_of_third_layer}
Anything associated to a given layer must be stored in a directory with the
same name as the layer, within the same directory where the environment
definition file is located. If we consider the my-app-environment
environment we used above then your directory tree should look like this:
.
├── my-app-environment.yaml
├── name_of_first_layer
│ ├── meta.yaml
│ └── resources.yaml
├── name_of_second_layer
│ ├── meta.json
│ └── meta.yaml
└── name_of_third_layer
├── resources.json.j2
└── resources.yaml.j2
A layer must contain at least two files:
meta.yaml
: Meta information about the layer such as a description, and layer parameters.resources.yaml
: Basically a CF template with the resources that the layer contains.
Those two files can also be in .json
format (meta.json
and
resources.json
). Or you can add the extension .j2
if you want the files to
be pre-processed with the Jinja2 template compiler.
Below an example of how a layer meta.yaml
may look like:
---
meta:
description:
Creates a VPC, that's it
parameters:
vpc_cidr:
description: The CIDR block of the VPC
value: 10.0.0.0/16
Above we declare only one layer parameter: vpc_cidr
. humilis
will make pass
that parameter to Jinja2 when compiling any template contained in the layer. So
the resources.yaml.j2
for that same layer may look like this:
---
resources:
VPC:
Type: "AWS::EC2::VPC"
Properties:
CidrBlock: {{ vpc_cidr }}
You can use references in your meta.yaml
files to refer to thing other than
resources within the same layer (to refer to resources within a layer you can
simply use Cloudformation's Ref or GetAtt functions).
Humilis references are used by setting the value of a layer parameter to a dict
that has a ref
key. Below an a meta.yaml
that refers to a resource (with
a logical name VPC
) that is contained in another layer (called vpc_layer
):
---
meta:
description:
Creates an EC2 instance in the vpc created by the vpc layer
dependencies:
- vpc
parameters:
vpc:
description: Physical ID of the VPC where the instance will be created
value:
ref:
parser: layer
parameters:
layer_name: vpc_layer
resource_name: VPC
Every reference must have a parser
key that identifies the parser that
should be used to parse the reference. There are also two optional keys:
-
parameters
: allows you to pass parameters to the reference parser. You can pass either named parameters (as a dict) or positional arguments (as a list). -
priority
: the parsing priority. Parameters with a lower value inpriority
will be parsed before parameters with a higher value. This allows some reference parsers to refer internally to other parameters within the same layer. For example, thelambda
parser, when parsing templated lambda code, it uses previously parsed layer parameters as template parameters.
More information on the reference parsers that are bundled with humilis below.
layer_resource
references allow you to refer to the physical ID of a resource
that is part of another layer.
Parameters:
layer_name
: The name of the layer you are referring toresource_name
: The logical name of the layer resource
Example:
Consider the following environment definition:
---
my-environment:
description:
Creates a VPC with a NAT in the public subnet
layers:
- {layer: vpc}
- {layer: nat}
Obviously the nat
layer that takes care of deploying the NAT in the public
subnet will need to know the physical ID of that subnet. You achieve this by
declaring a layer_resource
reference in the meta.yaml
for the nat
layer:
---
meta:
description:
Creates a managed NAT in the public subnet of the NAT layer
parameters:
subnet_id:
description:
The physical ID of the subnet where the NAT will be placed
value:
ref:
parser: layer_resource
parameters:
layer_name: vpc
# The logical name of the subnet in the vpc layer
resource_name: PublicSubnet
When parsing meta.yaml
humilis will replace this:
ref:
parser: layer_resource
parameters:
layer_name: vpc
# The logical name of the subnet in the vpc layer
resource_name: PublicSubnet
with the physical ID you need (something like subnet-bafa90cd
). You can then
use this physical ID in the resources.yaml.j2
section of the nat
layer:
{# Pseudo-content of layers/nat/resources.yaml.j2 #}
resources:
{# An Elastic IP reservation that will be associated to the NAT #}
NatEip:
Type: 'AWS::EC2::EIP'
Properties: {}
{# Custom resource deploying the NAT #}
NatGateway:
Type: 'Custom::NatGateway',
Properties:
{# The ARN of the Lambda function backing the custom resource #}
ServiceToken: 'arn:aws:lambda:eu-west-1:XXXX:function:CreateNatGateway'
{# Here we use the subnet_id reference defined in meta.yaml #}
SubnetId: {{subnet_id}}
AllocationId:
Ref: NatEip
environment_output
references allow you to refer to resources that belong
to other humilis environments.
Parameters:
environment_name
: The name of the environment you are referring tolayer_name
: The name of the layer you are referring toresource_name
: The logical name of the layer resource
layer_output
references allow you to refer to outputs produced by another
layer.
Parameters:
layer_name
: The name of the layer you are referring tooutput_name
: The logical name of the output parameter
In general you should prefer using layer_output
references over
layer_resource
references. The output parameters produced by a layer define
an informal layer interface that is more likely to remain constant than the
logical names of resources within a layer.
boto3
references define arbitrary calls to boto3facade. The
latter is just a simpler facade interface on top of boto3.
Parameters:
service
: The AWS service, e.g.ec2
orcloudformation
. Note that only only AWS services that have a facade in boto3facade are supported.call
: The corresponding facade method, e.g.get_ami_by_name
. The value of this parameter must be a dictionary with amethod
key (the name of the facade method to invoke) and an optionalargs
key (the parameters to pass to the facade method). Best to look at the example below to understand how this works.output_attribute
: Optional. If provided the reference parser will return the value of this attribute from the object returned by the facade method.
Below an example of a layer that uses a boto3
reference:
---
meta:
description:
Creates an EC2 instance using a named AMI
# More stuff omitted for brevity
ami:
description: The AMI to use when launching the EC2 instance
value:
ref:
parser: boto3
parameters:
service: ec2
call:
method: get_ami_by_name
args:
- test-ami
output_attribute: id
humilis
will parse the reference using this code:
# Import the Ec2 facade
from boto3facade.ec2 import Ec2
# Create a facade object
ec2_facade = Ec2()
# Make the call
ami = ec2_facade.get_ami_by_name('test-ami')
# Extract the requested attribute
ref_value = ami.id
file
references allow you to refer to a local file. The file will be uploaded
to S3 and the reference will evaluate to the corresponding S3 path.
Parameters:
path
: The path to the file, relative to the layer root directory.
lambda
references allow you to refer to some Python code in your local
machine. If your code follows some simple conventions humilis
will take care
of building a deployment package for you, uploading it
to S3, and the reference will evaluate to the S3 path of the deployment
package.
Parameters:
-
path
: Path to either a completely self-contained.py
file, or to the root directory of your lambda code. In the latter case your code needs to follow some simple conventions for this to work. More information below. -
dependencies
: A list of dependencies to be included in the Lambda deployment package. Dependencies may be either pip installable packages, or paths to local Python packages or modules, or paths to localrequirements
files.
Example:
ref:
parser: lambda
parameters:
# Path to the root directory containing your lambda code
path: dummy_function
dependencies:
# The Lambda code requires Pypi's pyyaml
- pyyaml
# It also requires a local package in this path
- mycode/mypkgdir
# And this local module
- mycode/mymodule.py
which will evaluate to a S3 path such as:
s3://[bucket_name]/[environment_name]/[stage_name]/[func_name]-[commithash].zip
Code conventions:
Following the example above, the contents of the layer responsible of deploying
the dummy_function
lambda may look like this:
.
├── dummy_function
│ ├── dummy_function.py
│ └── setup.py
├── meta.yaml
├── outputs.yaml.j2
└── resources.yaml.j2
Basically all your code needs to be included under directory dummy_function
.
In this case there is only one file: dummy_function.py
. External dependencies
need to be specified in your setup.py
.
secret
references retrieve a secret using Python's keyring module.
Parameters:
-
service
: The name of the service the secret is associated to. -
key
: The key (e.g. the username) that identifies the secret.
Example:
ref:
parser: secret
parameters: {"service": "mysqldb", "key": "adminuser"}
Humilis defines the following custom Jinja2 filters:
uuid
: A random UUID. Example:{{''|uuid}}
.password(length=8)
: A random password. Example:{{10|password}}
.