Skip to content
cloud architecture bricks
Python Other
  1. Python 99.9%
  2. Other 0.1%
Branch: master
Clone or download
Latest commit 907f1b8 Dec 6, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.devcontainer
.vscode Batman! (this commit has no parents) Dec 2, 2019
assets doc: link to deploy after install Dec 6, 2019
icons/aws Batman! (this commit has no parents) Dec 2, 2019
specs/aws chore: update cfn specs to 10.0.0 Dec 6, 2019
src fix: invalid custom resource image Dec 6, 2019
tests Batman! (this commit has no parents) Dec 2, 2019
.editorconfig Batman! (this commit has no parents) Dec 2, 2019
.gitattributes Batman! (this commit has no parents) Dec 2, 2019
.gitignore Batman! (this commit has no parents) Dec 2, 2019
.prettierignore Batman! (this commit has no parents) Dec 2, 2019
.prettierrc Batman! (this commit has no parents) Dec 2, 2019
LICENSE Batman! (this commit has no parents) Dec 2, 2019
Makefile Batman! (this commit has no parents) Dec 2, 2019
README.md
poetry.lock chore: update python dependencies Dec 6, 2019
poetry.toml Batman! (this commit has no parents) Dec 2, 2019
pyproject.toml chore(release): 0.1.1 Dec 6, 2019
setup.cfg Batman! (this commit has no parents) Dec 2, 2019

README.md

ic

Status License

ic is an open-source infrastructure as code development kit to describe your cloud architecture in a predictable, repeatable, and composable way using the Python programming language and AWS CloudFormation. It provides an isomorphism and eliminates the impedance mismatch between your source code, the intermediary Abstract Resource Tree, the low-level AWS CloudFormation template, and your deployed cloud resources.


ic, at its heart, can be assimilated to a restricted Python interpreter, which, given a set of source files, performs some transformations, and produces a human readable AWS CloudFormation template. As it has no opinion on how you have to deploy your infrastructure, it integrates seamlessly in your existing CI/CD pipeline.

In ic, everything is a resource, and a resource can either be transpiled to an AWS CloudFormation stack or be composed into a higher-level abstraction. You can consider your whole infrastructure as a nested tree of an arbitrary level. Plus, as ic resources are pure Python, you are free to partition, compose, and share any sub-tree using your favorite Python tools and package repositories.

ic comes with a purpose-built type system to provide you with enhanced code completion and feedback from within your IDE. It also extracts metadata directly from the official AWS CloudFormation Resource Specification so that you always have access to the latest AWS resources.

Table of Contents

Installation

Please note, as ic is pure Python 3.8, you can use any custom setup by installing the icdotdev package from PyPI.

The easiest way to play with ic is using Visual Studio Code with a Development Container, which means you need to have a working Docker installation. We put together everything you need to start crafting your first infrastructure brick.

  1. Download and extract the starter kit archive:

    mkdir brick; cd brick
    curl -L https://github.com/icdotdev/starter/archive/master.tar.gz | tar -xzv --strip-components=1
  2. Prepare your development container Docker image:

    cd .devcontainer; make; cd -
  3. Open the starter kit with Visual Studio Code and click "Reopen in Container" when prompted:

    code .

From here, you can either continue reading what follows or get a quick glimpse by deploying the starter kit.

Example

The below example (extracted from the starter kit) generates an AWS CloudFormation stack for an Amazon API Gateway backed by an AWS Lambda function. It produces a simple "Hello, World!" message.

# src/brick/__init__.py

from icaws import (
    AWSStr,
    aws_account_id,
    aws_apigateway,
    aws_iam,
    aws_lambda,
    aws_logs,
    aws_region,
    awsasset,
    awsdeferred,
    awsformat,
    awslink,
)
from iccore import Union, resource


@resource
def hello_world(name: str) -> AWSStr:
    return endpoint("endpoint", handler("handler"))


@resource
def handler(name: str) -> aws_lambda.Function:
    role = aws_iam.Role(
        "role",
        AssumeRolePolicyDocument=dict(
            Version="2012-10-17",
            Statement=dict(
                Effect="Allow",
                Principal=dict(Service="lambda.amazonaws.com"),
                Action="sts:AssumeRole",
            ),
        ),
        ManagedPolicyArns=[
            "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
        ],
    )
    function = aws_lambda.Function(
        "function",
        Runtime="nodejs10.x",
        Code=aws_lambda.Function.CodeType(ZipFile=awsasset("handler.js").text),
        Handler="index.handle",
        Role=role.Arn,
    )
    log_group = aws_logs.LogGroup(
        "log_group",
        LogGroupName=awsformat("/aws/lambda/{function}", function=function.Ref),
    )
    return function


@resource
def endpoint(
    name: str,
    handler: aws_lambda.Function,
    stage: Union[str, AWSStr] = awsdeferred("dev"),
) -> AWSStr:
    api = aws_apigateway.RestApi("api", Name="hello_world")
    method = aws_apigateway.Method(
        "method_get",
        RestApiId=api.Ref,
        ResourceId=api.RootResourceId,
        HttpMethod="GET",
        AuthorizationType="NONE",
        Integration=aws_apigateway.Method.IntegrationType(
            Type="AWS_PROXY",
            IntegrationHttpMethod="POST",
            Uri=awsformat(
                "arn:aws:apigateway:{region}:lambda:path/2015-03-31/functions/{function}/invocations",
                region=aws_region,
                function=handler.Arn,
            ),
        ),
    )
    permission = aws_lambda.Permission(
        "permission_invoke",
        FunctionName=handler.Arn,
        Action="lambda:InvokeFunction",
        Principal="apigateway.amazonaws.com",
        SourceArn=awsformat(
            "arn:aws:execute-api:{region}:{account}:{api}/*/GET/",
            region=aws_region,
            account=aws_account_id,
            api=api.Ref,
        ),
    )
    deployment = aws_apigateway.Deployment(
        "deployment", RestApiId=api.Ref, StageName=stage
    )
    awslink(deployment, method)
    url = awsformat(
        "https://{api}.execute-api.{region}.amazonaws.com/{stage}",
        api=api.Ref,
        region=aws_region,
        stage=stage,
    )
    return url
Generated AWS CloudFormation template
# Code generated by ic; DO NOT EDIT.
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  ICS3Bucket:
    Type: String
    AllowedPattern: .+
  ICS3Prefix:
    Type: String
    Default: ""
  EndpointStageJ2VIGA5D:
    Type: String
    Default: dev
Resources:
  EndpointApiFDSEHVHU:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: hello_world
  EndpointDeploymentGW2JFGSI:
    Type: AWS::ApiGateway::Deployment
    DependsOn:
      - EndpointMethodGetG32CKNG4
    Properties:
      RestApiId: !Ref EndpointApiFDSEHVHU
      StageName: !Ref EndpointStageJ2VIGA5D
  EndpointMethodGetG32CKNG4:
    Type: AWS::ApiGateway::Method
    Properties:
      AuthorizationType: NONE
      HttpMethod: GET
      Integration:
        Type: AWS_PROXY
        IntegrationHttpMethod: POST
        Uri:
          !Sub [
            "arn:aws:apigateway:${A0}:lambda:path/2015-03-31/functions/${A1}/invocations",
            { A0: !Ref "AWS::Region", A1: !GetAtt HandlerFunctionY3NGR3AL.Arn },
          ]
      ResourceId: !GetAtt EndpointApiFDSEHVHU.RootResourceId
      RestApiId: !Ref EndpointApiFDSEHVHU
  EndpointPermissionInvokeE7KNPQ2V:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt HandlerFunctionY3NGR3AL.Arn
      Principal: apigateway.amazonaws.com
      SourceArn:
        !Sub [
          "arn:aws:execute-api:${A0}:${A1}:${A2}/*/GET/",
          {
            A0: !Ref "AWS::Region",
            A1: !Ref "AWS::AccountId",
            A2: !Ref EndpointApiFDSEHVHU,
          },
        ]
  HandlerFunctionY3NGR3AL:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
          exports.handle = async () => ({
            statusCode: 200,
            body: "Hello, World!"
          });
      Handler: index.handle
      Role: !GetAtt HandlerRoleYQATF6FD.Arn
      Runtime: nodejs10.x
  HandlerLogGroupO5BT5OXX:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub ["/aws/lambda/${A0}", A0: !Ref HandlerFunctionY3NGR3AL]
  HandlerRoleYQATF6FD:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          Action: sts:AssumeRole
          Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Outputs:
  ID:
    Value: 270785f8c07d83cfbbc17db41ed20b43ca57a810
  Value:
    Value: !Sub
      - '"${A0}"'
      - A0:
          !Sub [
            "https://${A0}.execute-api.${A1}.amazonaws.com/${A2}",
            {
              A0: !Ref EndpointApiFDSEHVHU,
              A1: !Ref "AWS::Region",
              A2: !Ref EndpointStageJ2VIGA5D,
            },
          ]

Usage

Prerequisites

ic relies heavily on your AWS CLI configuration for anything related to AWS credentials and configs, which means you need to have a working AWS CLI installation. In particular, before performing any ic specific command, you need to select an AWS profile and an AWS region by exporting the following environment variables, according to your preferences:

export AWS_DEFAULT_PROFILE=sandbox
export AWS_DEFAULT_REGION=us-east-1

Note that if you are using the starter kit, then proceed as follows:

aws_config sandbox us-east-1

Authoring

An infrastructure brick starts as simple as a regular Python module containing at least one abstract resource (a node in the tree) and one concrete resource (a leaf in the tree).
Each node must have at least one positional argument, its name. And you must provide a name upon instantiation of each node.

You are free to create as many resources (both as function or classes) as you want, along with as many regular functions or classes for auxiliary purposes. However, to guarantee predictability and repeatability, some restrictions apply. As a general rule of thumb, anything that introduces any randomness or that is difficult to grasp for an absolute beginner in Python is rejected by the interpreter.
In Particular, all filesystem accesses, standard Python modules imports, and other features like threading, exceptions, magic methods, etc. are prohibited.

Even though types are optional, we advocate using them right from the beginning and everywhere.

from iccore import resource
from icaws import aws_lambda

@resource
def hello_world(name: str) -> None:
    aws_lambda.Function("my_awesome_function", ...)

Transpiling

To generate the low-level AWS CloudFormation template, use the ic CLI dump command and provide your Python module and its main resource. ic uses the standard Python import machinery to load your module. You are free to proceed as you please to structure your project and its dependencies. Note, that you cannot provide a relative module but your current working directory is automatically added to the Python modules search path.

In case you have any additional keyword arguments for your main resource, you can provide a JSON or a YAML file to the command. This file must contain the exact set of key and value pairs as required by your resource.

$ ic dump brick:hello_world config.yaml

The dump command creates a build directory inside your current working directory.

build/
├── config.json      # The config you have provided to the command
├── metadata.json    # Metadata for the internal usage of ic utils
├── source.zip       # All the source files used to generate the template
└── template.yaml    # The AWS CloudFormation template

To benefit from the isomorphism created by ic between your Python source code and your real infrastructure, you need to upload the entire content of the build directory to Amazon S3.

The ic CLI upload command looks for the following keys in your previously selected AWS profile, uploads the build artifacts, and outputs the URL of the AWS CloudFormation template.

# ~/.aws/config
[profile sandbox]
ic_s3_bucket = ic-deployment-bucket-1234
# ic_s3_prefix = foo/bar-
$ ic upload
upload template.yaml
upload config.json
upload metadata.json
upload source.zip
https://ic-deployment-bucket-1234.s3.amazonaws.com/instances/abcd7890/template.yaml

From here, you are free to use whatever tool you want to deploy your stack.

$ aws cloudformation deploy \
    --capabilities CAPABILITY_IAM \
    --template-file build/template.yaml \
    --stack-name hello-world

Debugging

The ic CLI tree command allows you to leverage the one-to-one relationship that exists between your Python source code, the Abstract Resource Tree, and your cloud resources.

You can uniformly visualize the resource tree of both your local Python module and your already deployed stack.

$ ic tree brick:hello_world     # The local module
$ ic tree hello-world           # The already deployed stack

root
├── endpoint
│   ├── api AWS::ApiGateway::RestApi
│   ├── deployment AWS::ApiGateway::Deployment
│   ├── method_get AWS::ApiGateway::Method
│   └── permission_invoke AWS::Lambda::Permission
└── handler
    ├── function AWS::Lambda::Function
    ├── log_group AWS::Logs::LogGroup
    └── role AWS::IAM::Role

You can directly access your stack resources in the AWS Management Console.

$ ic tree hello-world --link

root
├── endpoint
│   ├── api https://console.aws.amazon.com/apigateway/home?region=us-east-1#/apis/abcd1234
│   ├── deployment
│   ├── method_get
│   └── permission_invoke
└── handler
    ├── function https://console.aws.amazon.com/lambda/home?region=us-east-1#functions/hello-world-HandlerFunctionY3NGR3AL-ABCD1234
    ├── log_group https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logStream:group=/aws/lambda/hello-world-HandlerFunctionY3NGR3AL-ABCD1234
    └── role https://console.aws.amazon.com/iam/home?#/roles/hello-world-HandlerRoleYQATF6FD-ABCD1234

And you can also inspect the source code line responsible for the instantiation of a given resource in your stack.

$ ic tree hello-world --inspect
root
├── endpoint /tmp/tmptvgadjvf/workspaces/starter/src/brick/__init__.py:30
│   ├── api /tmp/tmptvgadjvf/workspaces/starter/src/brick/__init__.py:69
│   ├── deployment /tmp/tmptvgadjvf/workspaces/starter/src/brick/__init__.py:98
│   ├── method_get /tmp/tmptvgadjvf/workspaces/starter/src/brick/__init__.py:70
│   └── permission_invoke /tmp/tmptvgadjvf/workspaces/starter/src/brick/__init__.py:86
└── handler /tmp/tmptvgadjvf/workspaces/starter/src/brick/__init__.py:30
    ├── function /tmp/tmptvgadjvf/workspaces/starter/src/brick/__init__.py:49
    ├── log_group /tmp/tmptvgadjvf/workspaces/starter/src/brick/__init__.py:56
    └── role /tmp/tmptvgadjvf/workspaces/starter/src/brick/__init__.py:35

Give a try to the ic CLI tree-beta command too!

Contributing

The primary purpose of this project is to provide the easiest way to build and debug cloud-native architectures while preserving predictability and repeatability. We are grateful to the community for any contribution. You may, for example, proof-read the documentation, submit bugs and fixes, provide improvements, discuss new axes of evolution or spread the word around you.

Thanks

We want to thank and express our gratitude to Ben Kehoe warmly. He spent a lot of his time giving us invaluable advice. His expertise and vision in the Infrastructure as Code domain account for a considerable part of the evolution of this project.

License

Copyright 2019 Farzad Senart and Lionel Suss. All rights reserved.

Unless otherwise stated, the source code of the project is released under the GNU Affero General Public License Version 3.
Please note, however, that all public interfaces subject to be embedded within your source code are released under the Apache License Version 2. Refer to the header of each file or to the LICENSE file present in the parent directory where appropriate.

You can’t perform that action at this time.